Oct 30 19:32:04.567868 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:32:04.567879 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:32:04.567886 ip-10-0-132-4 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:32:04.568426 ip-10-0-132-4 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:32:14.817070 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:32:14.817089 ip-10-0-132-4 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 40c49548cb194cc885d22793bbcebe88 -- Oct 30 19:34:20.800660 ip-10-0-132-4 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:34:21.186491 ip-10-0-132-4 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:21.186491 ip-10-0-132-4 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:34:21.186491 ip-10-0-132-4 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:21.186491 ip-10-0-132-4 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:21.191802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191705 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.191802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191799 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.191802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191804 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.191802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191806 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.191802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191809 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191813 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191816 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191818 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191821 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191823 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191826 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191829 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191832 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191834 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191837 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191839 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191842 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191844 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191847 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191849 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191853 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191855 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191859 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191861 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.191946 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191864 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191866 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191869 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191871 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191874 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191876 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191879 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191881 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191884 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191886 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191889 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191892 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191894 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191897 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191900 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191903 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191906 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191909 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191911 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191914 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.192414 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191916 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191919 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191921 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191924 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191926 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191929 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191931 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191933 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191936 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191938 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191941 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191944 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191946 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191949 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191952 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191954 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191957 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191959 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191962 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191964 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.192905 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191967 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191971 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191975 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191978 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191981 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191984 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191988 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191990 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191993 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191995 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.191998 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192000 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192003 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192005 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192008 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192010 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192013 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192015 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192018 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.193360 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192020 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192023 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192025 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192028 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192031 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192034 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192037 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192040 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192042 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192045 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192048 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192050 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192053 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192055 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192058 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192060 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192063 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192065 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192068 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.193823 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192072 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192077 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192081 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192085 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192939 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192947 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192951 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192954 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192958 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192961 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192963 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192966 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192968 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192971 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192973 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192976 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192979 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192981 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192984 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.194267 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192987 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192990 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192992 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.192998 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193001 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193003 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193005 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193008 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193011 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193013 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193015 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193018 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193020 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193023 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193026 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193029 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193032 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193034 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193037 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.194734 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193040 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193042 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193045 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193047 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193050 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193052 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193055 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193057 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193059 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193062 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193064 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193067 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193069 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193072 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193074 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193077 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193082 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193085 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193087 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.195175 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193091 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193095 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193098 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193101 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193103 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193106 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193109 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193112 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193115 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193117 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193120 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193122 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193125 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193128 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193131 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193133 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193137 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193140 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193143 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.195624 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193146 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193149 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193152 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193155 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193158 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193160 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193163 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193165 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193168 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193172 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193175 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193178 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193181 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193184 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193186 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193189 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193191 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193194 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193196 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193199 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.196064 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193201 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193204 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193206 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193209 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193211 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193214 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193216 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193219 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193221 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193224 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193226 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193230 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193232 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193235 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193776 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193783 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193787 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193790 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193793 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193795 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.196559 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193798 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193802 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193807 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193810 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193813 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193816 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193819 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193823 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193828 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193831 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193834 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193837 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193839 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193842 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193844 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193847 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193850 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193852 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193855 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193857 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.197010 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193860 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193862 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193865 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193867 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193870 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193873 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193875 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193877 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193882 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193885 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193889 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193893 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193896 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193899 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193902 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193905 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193907 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193910 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193913 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.197562 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193915 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193917 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193920 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193923 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193925 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193927 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193930 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193932 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193934 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193937 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193939 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193942 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193944 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193947 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193949 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193951 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193954 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193956 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193958 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.198042 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193961 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193963 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193967 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193969 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193972 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193974 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193977 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193979 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193982 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193985 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193988 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193990 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193993 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193995 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.193998 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194000 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194002 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194005 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194007 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194009 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.198503 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194012 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194014 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194017 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194019 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194021 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194023 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194026 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194028 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194031 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194033 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194036 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194038 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194040 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194044 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194048 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194050 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194052 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194055 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194057 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.198960 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194059 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194062 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.194064 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.194975 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.194985 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.194993 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.194998 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195003 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195006 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195010 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195014 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195018 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195021 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195025 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195028 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195031 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195034 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195037 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195039 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195042 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195045 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195049 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195052 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195055 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:34:21.199438 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195058 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195061 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195065 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195069 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195072 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195075 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195078 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195081 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195085 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195088 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195091 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195095 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195099 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195102 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195105 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195108 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195111 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195119 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195122 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195126 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195129 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195132 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195136 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195139 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195142 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:34:21.199964 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195145 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195148 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195151 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195154 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195157 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195160 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195163 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195166 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195171 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195173 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195177 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195181 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195184 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195187 2572 flags.go:64] FLAG: --help="false" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195190 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195194 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195199 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195202 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195205 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195209 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195213 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195216 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195219 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:34:21.200586 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195222 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195224 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195227 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195230 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195233 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195236 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195239 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195242 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195245 2572 flags.go:64] FLAG: --lock-file="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195248 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195251 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195254 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195259 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195262 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195265 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195268 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195271 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195275 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195277 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195280 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195286 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195288 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195292 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195295 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195298 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:34:21.201120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195303 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195305 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195308 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195311 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195314 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195323 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195326 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195329 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195332 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195335 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195337 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195341 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195343 2572 flags.go:64] FLAG: --port="10250" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195347 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195349 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08cb5a7224d0c2cf8" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195353 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195356 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195359 2572 flags.go:64] FLAG: --register-node="true" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195374 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195380 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195384 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195388 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195392 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195396 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195400 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195403 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:34:21.201766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195406 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195409 2572 flags.go:64] FLAG: --runonce="false" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195412 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195415 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195418 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195421 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195426 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195429 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195432 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195435 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195438 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195441 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195445 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195448 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195451 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195454 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195457 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195462 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195465 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195468 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195471 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195474 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195477 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195480 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195483 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:34:21.202769 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195486 2572 flags.go:64] FLAG: --v="2" Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195491 2572 flags.go:64] FLAG: --version="false" Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195495 2572 flags.go:64] FLAG: --vmodule="" Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195499 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195503 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195664 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195669 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195674 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195676 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195679 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195681 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195684 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195686 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195690 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195693 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195695 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195698 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195700 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195703 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195705 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.203568 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195709 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195711 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195714 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195717 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195719 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195722 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195724 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195727 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195729 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195732 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195734 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195738 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195742 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195746 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195748 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195751 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195754 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195757 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195759 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.204041 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195763 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195766 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195768 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195771 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195773 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195776 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195779 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195782 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195784 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195787 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195789 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195792 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195794 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195797 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195800 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195802 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195805 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195808 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195810 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195812 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.204553 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195815 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195817 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195819 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195822 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195824 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195827 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195829 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195832 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195834 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195837 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195839 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195842 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195845 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195849 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195852 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195855 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195858 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195860 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195864 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195867 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.205011 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195869 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195872 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195874 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195877 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195879 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195882 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195884 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195887 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195890 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195892 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195895 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195897 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195900 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195902 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195904 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195907 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195910 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195912 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195915 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195917 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.205486 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195919 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195922 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195924 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195927 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195930 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195933 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195935 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195938 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195940 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195943 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195946 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.195949 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.195974 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196027 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196032 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196035 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.205953 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196038 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196041 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196044 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196047 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196051 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196053 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196056 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196059 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196061 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196064 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196066 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196070 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196074 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196077 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196079 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196082 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196085 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196087 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196089 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196092 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.206332 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196096 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196098 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196101 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196103 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196106 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196108 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196110 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196114 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196117 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196119 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196122 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196124 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196127 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196129 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196132 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196134 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196137 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196140 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196142 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196145 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.206802 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196147 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196150 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196152 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196154 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196157 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196159 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196161 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196164 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196166 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196169 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196172 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196174 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196177 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196180 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196182 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196185 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196187 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196190 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196192 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.207263 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196195 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196198 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196201 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196203 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196206 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196208 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196211 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196214 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196216 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196219 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196222 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196225 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196227 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196230 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196232 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196235 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196237 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196241 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196244 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.207723 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196247 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196251 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196253 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196256 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196259 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196261 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196265 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196267 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196269 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196272 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196274 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196277 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196279 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196283 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196286 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196288 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196291 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196293 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196296 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.208150 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196298 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196301 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196303 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196306 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196309 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.196312 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.204127 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.204150 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.204167 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.204172 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204256 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204262 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204265 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204269 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204272 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204275 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204278 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204281 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204285 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.208716 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204288 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204292 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204296 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204299 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204303 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204306 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204308 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204311 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204313 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204316 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204319 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204321 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204324 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204326 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204329 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204332 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204335 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204338 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204341 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204344 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.209147 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204347 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204349 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204352 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204355 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204357 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204360 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204382 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204384 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204387 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204390 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204393 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204396 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204398 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204401 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204404 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204407 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204410 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204413 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204418 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.209625 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204422 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204425 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204428 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204430 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204433 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204435 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204438 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204440 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204443 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204445 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204448 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204451 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204454 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204457 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204459 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204462 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204464 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204467 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204470 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204473 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.210054 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204475 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204477 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204480 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204483 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204486 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204488 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204491 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204493 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204496 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204499 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204501 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204505 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204507 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204510 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204512 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204515 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204518 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204520 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204522 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.210532 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204525 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204528 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204530 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204533 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204536 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204538 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204541 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204543 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204546 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204548 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204551 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204553 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204556 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204559 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204561 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204565 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204567 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204570 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.210956 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204573 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.204605 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204753 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204757 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204760 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204763 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204766 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204770 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204773 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204775 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204777 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204780 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204782 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204785 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204787 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204789 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.211376 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204792 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204795 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204798 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204800 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204803 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204805 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204810 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204814 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204817 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204820 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204822 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204825 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204827 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204830 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204832 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204835 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204837 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204840 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204842 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.211790 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204844 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204847 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204850 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204852 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204854 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204857 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204860 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204862 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204865 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204867 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204870 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204872 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204874 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204877 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204880 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204883 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204885 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204888 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204890 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204893 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.212234 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204896 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204901 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204903 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204906 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204908 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204911 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204913 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204916 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204919 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204921 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204923 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204926 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204928 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204931 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204933 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204935 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204938 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204941 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.212701 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204944 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204947 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204949 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204951 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204954 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204956 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204959 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204961 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204964 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204966 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204969 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204971 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204976 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204979 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204981 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204984 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204986 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204989 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204992 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204994 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.213151 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204997 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.204999 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205002 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205005 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205009 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205011 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205014 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205017 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205019 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205022 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205024 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205027 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205030 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205033 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.205060 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:21.213616 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205092 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205097 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205100 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205103 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205106 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205109 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205111 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205114 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205116 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205120 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205123 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205125 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205128 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205130 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205133 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205135 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205138 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205141 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205143 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205145 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:21.213973 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205148 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205150 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205153 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205155 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205157 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205160 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205162 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205165 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205168 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205170 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205174 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205176 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205179 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205181 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205184 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205186 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205190 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205193 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205196 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:21.214470 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205199 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205201 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205205 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205208 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205210 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205213 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205216 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205218 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205220 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205223 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205225 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205228 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205230 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205232 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205235 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205238 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205240 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205243 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205245 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205247 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:21.214993 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205250 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205254 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205257 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205259 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205262 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205265 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205267 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205270 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205274 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205277 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205280 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205283 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205285 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205288 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205292 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205294 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205297 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205299 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205302 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:21.215527 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205304 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205306 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205309 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205312 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205315 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205318 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205320 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205323 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205325 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205328 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205330 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205332 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205335 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205337 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205341 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205343 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205346 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205349 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205352 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205355 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:21.216029 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205357 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205360 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205376 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205380 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205382 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205385 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205387 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:21.205392 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.205460 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.207945 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.208818 2572 server.go:1014] "Starting client certificate rotation" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.208921 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.208963 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.210667 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:34:21.216641 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.211825 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:34:21.217906 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.217892 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:34:21.220325 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.220309 2572 log.go:25] "Validated CRI v1 image API" Oct 30 19:34:21.221093 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.221062 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:34:21.226425 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.226391 2572 fs.go:135] Filesystem UUIDs: map[0c4b3811-bf74-40ee-bd1d-6a26a3e7ae18:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ff6df988-3d6a-48a4-84df-9ece43f90eb4:/dev/nvme0n1p4] Oct 30 19:34:21.226545 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.226423 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:34:21.232890 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.232761 2572 manager.go:217] Machine: {Timestamp:2026-10-30 19:34:21.231128114 +0000 UTC m=+0.315392546 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3194046 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be418399c6e50cde7cc0dcacd0761 SystemUUID:ec2be418-399c-6e50-cde7-cc0dcacd0761 BootID:40c49548-cb19-4cc8-85d2-2793bbcebe88 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:d4:22:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:d4:22:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:93:e7:61:dc:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:34:21.232890 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.232882 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:34:21.233020 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.233013 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:34:21.236120 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236087 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:34:21.236295 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236121 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:34:21.236388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236300 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:34:21.236388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236312 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:34:21.236388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236350 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:34:21.236388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.236385 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:34:21.237210 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.237197 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:34:21.240185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240169 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:34:21.240328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240316 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:34:21.240936 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240925 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:34:21.240987 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240943 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:34:21.240987 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240957 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:34:21.240987 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240971 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:34:21.240987 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.240983 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:34:21.241382 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.241354 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:34:21.243625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.243607 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:34:21.248111 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.248086 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:34:21.248938 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.248924 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:34:21.248980 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.248956 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:34:21.249138 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249127 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:34:21.249167 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249144 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:34:21.249167 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249153 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:34:21.249167 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249161 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:34:21.249167 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249166 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:34:21.249257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249173 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:34:21.249889 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249879 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:34:21.249926 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249890 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:34:21.249926 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249897 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:34:21.249926 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249904 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:34:21.249926 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249913 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:34:21.249926 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.249922 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:34:21.251216 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.251201 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:34:21.251216 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.251215 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:34:21.255134 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.255121 2572 server.go:1263] "Started kubelet" Oct 30 19:34:21.255242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.255186 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:34:21.255278 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.255234 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:34:21.255318 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.255302 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:34:21.255963 ip-10-0-132-4 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:34:21.256404 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.256383 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:34:21.257962 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.257944 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:34:21.260684 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.260554 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:34:21.262078 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.262053 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:34:21.262591 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.262574 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:34:21.262703 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.262686 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.263345 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263328 2572 factory.go:55] Registering systemd factory Oct 30 19:34:21.263345 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263344 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:34:21.263543 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263525 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:34:21.263543 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263544 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263523 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263584 2572 factory.go:153] Registering CRI-O factory Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263598 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263627 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263635 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263645 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263664 2572 factory.go:103] Registering Raw factory Oct 30 19:34:21.263683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.263678 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 19:34:21.264562 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.264548 2572 manager.go:319] Starting recovery of all containers Oct 30 19:34:21.275291 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.275270 2572 manager.go:324] Recovery completed Oct 30 19:34:21.279411 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.279393 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.281892 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.281854 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.281952 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.281910 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.281952 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.281921 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.282476 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.282457 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:34:21.282476 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.282473 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:34:21.282580 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.282491 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:34:21.282580 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.282562 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.283245 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283231 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.283288 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283259 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.283288 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283271 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.283906 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283894 2572 policy_none.go:50] "Start" Oct 30 19:34:21.283952 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283911 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:34:21.283952 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.283925 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:34:21.285023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.285012 2572 policy_none.go:44] "Start" Oct 30 19:34:21.301939 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.301913 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:34:21.302057 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.301977 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:34:21.307289 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.307266 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:34:21.313465 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.313439 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:34:21.313465 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.313446 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:34:21.313465 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.313442 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:34:21.313730 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.312782 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e365aae3408f4d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-10-30 19:34:21.255094093 +0000 UTC m=+0.339358525,LastTimestamp:2026-10-30 19:34:21.255094093 +0000 UTC m=+0.339358525,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}" Oct 30 19:34:21.314187 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.314157 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:34:21.316749 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.316727 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 19:34:21.316843 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.316765 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:34:21.316843 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.316776 2572 server.go:87] "Starting device plugin registration server" Oct 30 19:34:21.317034 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.317021 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.317037 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.317168 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.317253 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.317261 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.317771 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:34:21.318052 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.317836 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.318460 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.318408 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8njzx" Oct 30 19:34:21.330980 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.330868 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8njzx" Oct 30 19:34:21.331175 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.331084 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e365aae4d979be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-10-30 19:34:21.281892798 +0000 UTC m=+0.366157230,LastTimestamp:2026-10-30 19:34:21.281892798 +0000 UTC m=+0.366157230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}" Oct 30 19:34:21.405545 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.405514 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:34:21.406911 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.406889 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:34:21.406911 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.406913 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:34:21.407028 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.406934 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:34:21.407028 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.406965 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:34:21.417600 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.417572 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.418548 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.418526 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.418647 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.418560 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.418647 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.418573 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.418647 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.418596 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.425083 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.425065 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:21.441442 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.441393 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.441442 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.441423 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-4.ec2.internal\": node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.497239 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.497217 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.507460 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.507420 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"] Oct 30 19:34:21.507536 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.507496 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.508509 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.508485 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.508611 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.508517 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.508611 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.508533 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.510432 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.510419 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.510603 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.510590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.510648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.510623 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.511155 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511139 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.511232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511161 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.511232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511170 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.511232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511219 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.511385 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511243 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.511385 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.511256 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.512938 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.512920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.513028 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.512960 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:21.513782 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.513762 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:21.513921 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.513804 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:21.513921 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.513817 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:21.543551 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.543524 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.546509 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.546490 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.565296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.565256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.565296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.565292 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.565492 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.565318 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.598029 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.597993 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.665661 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.665756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.665756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.665818 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665761 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.665818 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665798 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.665879 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.665838 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.698492 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.698436 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.799383 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.799337 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:21.847477 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.847446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.850880 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:21.850861 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:21.899613 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:21.899426 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:22.000065 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:22.000017 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:22.102588 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:22.102548 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:22.202807 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:22.202774 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:22.209459 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.209426 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:34:22.209597 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.209577 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:34:22.303065 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:22.302851 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:34:22.330408 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.330361 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:22.334718 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.334671 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:29:21 +0000 UTC" deadline="2028-07-12 15:11:11.322398628 +0000 UTC" Oct 30 19:34:22.334718 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.334714 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14899h36m48.987689529s" Oct 30 19:34:22.343485 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.343456 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:22.363100 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.363071 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:34:22.388482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.388458 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:22.390109 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.390087 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:34:22.419550 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.419521 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:22.426081 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:22.426046 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c9af704a53c4c9b04653e0d8d8d8f.slice/crio-db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e WatchSource:0}: Error finding container db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e: Status 404 returned error can't find the container with id db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e Oct 30 19:34:22.434149 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.434130 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:34:22.437655 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:22.437632 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dff8c1e3d20edc13c5947130e8a5c14.slice/crio-fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb WatchSource:0}: Error finding container fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb: Status 404 returned error can't find the container with id fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb Oct 30 19:34:22.851726 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:22.851692 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:23.242921 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.242542 2572 apiserver.go:52] "Watching apiserver" Oct 30 19:34:23.263671 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.263570 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:34:23.276760 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.276715 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:34:23.277161 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.277125 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-ftqg8","openshift-multus/multus-additional-cni-plugins-wksc9","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-multus/multus-t5d7p","kube-system/global-pull-secret-syncer-zzgh7","kube-system/konnectivity-agent-l7lfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","openshift-network-diagnostics/network-check-target-xspfd","openshift-ovn-kubernetes/ovnkube-node-k5fx8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws","openshift-cluster-node-tuning-operator/tuned-s9qdt","openshift-dns/node-resolver-hmsqs","openshift-multus/network-metrics-daemon-8lk6r","openshift-network-operator/iptables-alerter-twm5d"] Oct 30 19:34:23.280007 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.279965 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.282325 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.282292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.285280 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.285255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.287583 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.287553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:23.287797 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.287642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:23.290591 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.290568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.293084 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.293061 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.293188 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.293138 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:23.296953 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.296555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.299408 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.299193 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.300574 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.300550 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.300736 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.300668 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:34:23.300824 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.300731 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.300953 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.300936 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\"" Oct 30 19:34:23.301050 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.301032 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.301371 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.301353 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:34:23.301888 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.301869 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.302111 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.302094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.302923 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.302899 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:34:23.303075 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.303057 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\"" Oct 30 19:34:23.305237 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.305192 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.308181 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.308160 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.308262 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.308247 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:23.308776 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.308728 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.309199 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.309179 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.309286 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.309217 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:34:23.309286 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.309247 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:34:23.309473 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.309453 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:34:23.310611 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.310588 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.310955 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.310935 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:34:23.311052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.311006 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\"" Oct 30 19:34:23.311052 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.311028 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\"" Oct 30 19:34:23.311162 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.311062 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:34:23.319073 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.319035 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:34:23.327016 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.326989 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8kldv" Oct 30 19:34:23.340057 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.340020 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8kldv" Oct 30 19:34:23.365950 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.365912 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:34:23.376343 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376301 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.376343 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376345 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.376587 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376425 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.376587 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.376587 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.376720 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.376720 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.376831 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.376831 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.376831 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.376831 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmbd\" (UniqueName: \"kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.376982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.377038 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg285\" (UniqueName: \"kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377213 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.377301 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377497 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.377706 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4z8q\" (UniqueName: \"kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw4f\" (UniqueName: \"kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdh6l\" (UniqueName: \"kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377952 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.377979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378061 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378184 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.378306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378302 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378335 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59jq\" (UniqueName: \"kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378582 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7gx\" (UniqueName: \"kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378779 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.378814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378814 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49cj\" (UniqueName: \"kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.378966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.379119 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379124 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379160 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvzt\" (UniqueName: \"kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.379328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.379221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.407377 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407342 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:34:23.407566 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407478 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\"" Oct 30 19:34:23.407566 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407488 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\"" Oct 30 19:34:23.407566 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407520 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\"" Oct 30 19:34:23.407683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407577 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\"" Oct 30 19:34:23.407683 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.407628 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\"" Oct 30 19:34:23.408437 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408351 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.408579 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408521 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.408891 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408871 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.409009 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408893 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.409009 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408941 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.409009 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.408988 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409031 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409098 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409164 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409032 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409100 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:34:23.409257 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.409235 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:34:23.411924 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.411872 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb"} Oct 30 19:34:23.413424 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.413343 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e"} Oct 30 19:34:23.445601 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.445565 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:23.480252 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49cj\" (UniqueName: \"kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.480461 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.480461 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.480461 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.480461 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480418 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.480625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480498 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.480625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.480625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.480625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480659 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvzt\" (UniqueName: \"kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.480766 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480851 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480959 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.480982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmbd\" (UniqueName: \"kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481023 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg285\" (UniqueName: \"kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481268 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481393 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.481362 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.481510 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.481450 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:23.482356 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.482287 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.482426 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.482408 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:23.981496689 +0000 UTC m=+3.065761140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:23.483128 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.483102 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.483521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.483621 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.483691 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.483872 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.483984 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.484046 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:23.984022937 +0000 UTC m=+3.068287373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484052 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484089 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484131 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484204 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484339 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484392 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484551 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.485391 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484720 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484731 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484793 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484842 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484937 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z8q\" (UniqueName: \"kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.484978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw4f\" (UniqueName: \"kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh6l\" (UniqueName: \"kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485013 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.486339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485218 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485278 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485329 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485361 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485494 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.487106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485604 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485656 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485662 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485663 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485708 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485735 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485760 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485830 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485837 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.485880 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.486708 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.486840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.486911 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487058 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.487761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59jq\" (UniqueName: \"kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7gx\" (UniqueName: \"kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.487957 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488098 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488384 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488457 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.488756 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.489576 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.488641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.489576 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489396 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.489576 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489531 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.489721 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489594 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.489773 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.489829 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489814 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.489877 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489865 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.489931 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.489887 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.491242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.490293 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.491242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.490620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.491242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.490876 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.491242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.491117 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.491242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.491126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.492070 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.492046 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.492781 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.492759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.493644 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.492915 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.493644 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.493262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.495505 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.495430 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.505334 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.505285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49cj\" (UniqueName: \"kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.509331 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.509294 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:23.509331 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.509332 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:23.509642 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.509349 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:23.509642 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.509541 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:24.009446323 +0000 UTC m=+3.093710769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:23.511605 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.511461 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59jq\" (UniqueName: \"kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.512263 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.512230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmbd\" (UniqueName: \"kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.513097 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.513065 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh6l\" (UniqueName: \"kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.513509 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.513148 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw4f\" (UniqueName: \"kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.513509 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.513359 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7gx\" (UniqueName: \"kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.513810 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.513757 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg285\" (UniqueName: \"kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.514881 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.514848 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvzt\" (UniqueName: \"kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.515548 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.515502 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z8q\" (UniqueName: \"kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.596684 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.596635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:34:23.610329 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.610299 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac523a2_773b_493f_bb76_8faa6f8c3bb1.slice/crio-d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4 WatchSource:0}: Error finding container d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4: Status 404 returned error can't find the container with id d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4 Oct 30 19:34:23.619822 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.619794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:34:23.628662 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.628515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:23.628662 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.628612 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:23.631510 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.631485 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t5d7p" Oct 30 19:34:23.632047 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.632019 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557d0d40_9bfa_4d88_9ee2_ac07635e7daa.slice/crio-dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f WatchSource:0}: Error finding container dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f: Status 404 returned error can't find the container with id dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f Oct 30 19:34:23.644203 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.644172 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908aaccc_1937_4ac9_8614_28365a9fb984.slice/crio-e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421 WatchSource:0}: Error finding container e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421: Status 404 returned error can't find the container with id e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421 Oct 30 19:34:23.644417 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.644367 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:34:23.648761 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.648508 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad1332e_70fd_49ce_b1c2_48e50110f29b.slice/crio-a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd WatchSource:0}: Error finding container a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd: Status 404 returned error can't find the container with id a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd Oct 30 19:34:23.648761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.648610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:34:23.660207 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.660179 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:34:23.661396 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.661360 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4009f08d_95f6_4e73_ba56_2740e4e55336.slice/crio-8385280db8d859edc2954b8eef37c259de4ddd1b6e28f217e113de19659495b1 WatchSource:0}: Error finding container 8385280db8d859edc2954b8eef37c259de4ddd1b6e28f217e113de19659495b1: Status 404 returned error can't find the container with id 8385280db8d859edc2954b8eef37c259de4ddd1b6e28f217e113de19659495b1 Oct 30 19:34:23.673860 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.673815 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58f03ba_4840_4237_88ee_82bf233ac315.slice/crio-fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699 WatchSource:0}: Error finding container fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699: Status 404 returned error can't find the container with id fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699 Oct 30 19:34:23.678979 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.678951 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:34:23.691380 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:34:23.691332 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5f60ac_4e68_4bb9_bd1f_d08bfbc20488.slice/crio-cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893 WatchSource:0}: Error finding container cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893: Status 404 returned error can't find the container with id cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893 Oct 30 19:34:23.993690 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.993651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:23.993690 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:23.993694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:23.993922 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.993829 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:23.993922 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.993901 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:24.993884649 +0000 UTC m=+4.078149084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:23.993997 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.993830 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:23.993997 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:23.993987 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:24.993970247 +0000 UTC m=+4.078234683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:24.094153 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.094108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:24.094398 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:24.094372 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:24.094475 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:24.094402 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:24.094475 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:24.094417 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:24.094618 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:24.094591 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.094569353 +0000 UTC m=+4.178833780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:24.344339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.344121 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:29:23 +0000 UTC" deadline="2028-06-04 08:37:22.733583223 +0000 UTC" Oct 30 19:34:24.344339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.344199 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13981h2m58.389389033s" Oct 30 19:34:24.409627 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.409057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:24.409627 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:24.409216 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:24.418431 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.418292 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftqg8" event={"ID":"3ac523a2-773b-493f-bb76-8faa6f8c3bb1","Type":"ContainerStarted","Data":"d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4"} Oct 30 19:34:24.420571 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.420464 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerStarted","Data":"cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893"} Oct 30 19:34:24.424058 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.423954 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" event={"ID":"4009f08d-95f6-4e73-ba56-2740e4e55336","Type":"ContainerStarted","Data":"8385280db8d859edc2954b8eef37c259de4ddd1b6e28f217e113de19659495b1"} Oct 30 19:34:24.426279 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.426102 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" event={"ID":"814f2828-2c09-40ea-b910-02df6029fffd","Type":"ContainerStarted","Data":"f17572b48b2653eaa77c916aee7eb8bcf5e833a2b4594c0b1ff9b6344408bc6f"} Oct 30 19:34:24.438650 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.438544 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t5d7p" event={"ID":"fad1332e-70fd-49ce-b1c2-48e50110f29b","Type":"ContainerStarted","Data":"a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd"} Oct 30 19:34:24.462059 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.461637 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421"} Oct 30 19:34:24.474754 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.474710 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"ea90a98b8fdfc8add1b448cc029d8020d02a4177d87936f4126db2406508cdb1"} Oct 30 19:34:24.479030 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.478988 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmsqs" event={"ID":"c58f03ba-4840-4237-88ee-82bf233ac315","Type":"ContainerStarted","Data":"fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699"} Oct 30 19:34:24.481115 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.481063 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7lfb" event={"ID":"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e","Type":"ContainerStarted","Data":"679637e378b0fdf60731b9585d4b264710905192bd769a250320d90269ea68ab"} Oct 30 19:34:24.496227 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:24.496190 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twm5d" event={"ID":"557d0d40-9bfa-4d88-9ee2-ac07635e7daa","Type":"ContainerStarted","Data":"dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f"} Oct 30 19:34:25.005735 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.005700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:25.005735 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.005740 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:25.006009 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.005945 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:25.006071 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.006011 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:27.005992858 +0000 UTC m=+6.090257281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:25.006466 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.006419 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:25.006571 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.006494 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:27.006477737 +0000 UTC m=+6.090742161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:25.107388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.107341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:25.107624 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.107604 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:25.107734 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.107633 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:25.107734 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.107647 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:25.107734 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.107713 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:27.107694208 +0000 UTC m=+6.191958632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:25.345255 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.345120 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:29:23 +0000 UTC" deadline="2028-06-11 18:16:59.92177673 +0000 UTC" Oct 30 19:34:25.345255 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.345170 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14158h42m34.576613067s" Oct 30 19:34:25.408627 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.408570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:25.408787 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.408694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:25.409404 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.409329 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:25.409511 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:25.409480 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:25.505493 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.505430 2572 generic.go:356] "Generic (PLEG): container finished" podID="5dff8c1e3d20edc13c5947130e8a5c14" containerID="112ff42866bd54eba2419081a3b93f49b5bacfdce5a496f0052298978116945d" exitCode=0 Oct 30 19:34:25.506595 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.506567 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerDied","Data":"112ff42866bd54eba2419081a3b93f49b5bacfdce5a496f0052298978116945d"} Oct 30 19:34:25.552094 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:25.552018 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" podStartSLOduration=3.551993142 podStartE2EDuration="3.551993142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:24.521061917 +0000 UTC m=+3.605326360" watchObservedRunningTime="2026-10-30 19:34:25.551993142 +0000 UTC m=+4.636257583" Oct 30 19:34:26.408136 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:26.408042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:26.412481 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:26.408210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:26.515857 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:26.515814 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"e6ff4aa892aaddca7da54639fb3560aefa1bd93229f69055c0705b8b87b6d43f"} Oct 30 19:34:26.561634 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:26.561330 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" podStartSLOduration=4.561311697 podStartE2EDuration="4.561311697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:26.560929915 +0000 UTC m=+5.645194361" watchObservedRunningTime="2026-10-30 19:34:26.561311697 +0000 UTC m=+5.645576143" Oct 30 19:34:27.024856 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.024764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:27.025024 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.024877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:27.025024 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.025005 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:27.025149 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.025073 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.025052431 +0000 UTC m=+10.109316866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:27.025537 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.025513 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:27.025642 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.025572 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.025557013 +0000 UTC m=+10.109821434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:27.125476 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.125428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:27.125662 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.125647 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:27.125718 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.125670 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:27.125718 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.125684 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:27.125816 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.125747 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.125726724 +0000 UTC m=+10.209991161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:27.409318 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.408750 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:27.409318 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.408766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:27.409318 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.408897 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:27.409318 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:27.409086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:27.524432 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.524251 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="7de76715ddcb647fdf1014965e1c5778adfa25fabdb775105f946f3a6deff249" exitCode=0 Oct 30 19:34:27.525650 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:27.525538 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"7de76715ddcb647fdf1014965e1c5778adfa25fabdb775105f946f3a6deff249"} Oct 30 19:34:28.407701 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:28.407664 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:28.407911 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:28.407820 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:28.530414 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:28.530374 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7lfb" event={"ID":"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e","Type":"ContainerStarted","Data":"14ab9ff7670ff39e8b10c7fb7e17a14afea523379efcc97069eca644d8b427d4"} Oct 30 19:34:29.409732 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.409404 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:29.409732 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:29.409501 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:29.409732 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.409553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:29.409732 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:29.409610 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:29.502921 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.502881 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:29.503727 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.503700 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:29.541876 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.541848 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftqg8" event={"ID":"3ac523a2-773b-493f-bb76-8faa6f8c3bb1","Type":"ContainerStarted","Data":"b26216a096fe69d8df987c047ea0354daa1c23a3d48749b1ef81d7f54a18df2a"} Oct 30 19:34:29.546824 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:29.545439 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l7lfb" podStartSLOduration=3.425825433 podStartE2EDuration="7.54542362s" totalImagesPullingTime="4.119598187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:28.58668889 +0000 UTC m=+7.670953335" watchObservedRunningTime="2026-10-30 19:34:29.54542362 +0000 UTC m=+8.629688061" Oct 30 19:34:30.407817 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:30.407740 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:30.407962 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:30.407894 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:30.547580 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:30.547538 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" event={"ID":"814f2828-2c09-40ea-b910-02df6029fffd","Type":"ContainerStarted","Data":"1cd3cccaa287927a9f5b4227f2dde55bfe5f9d3bd676d4886353847da07b6160"} Oct 30 19:34:30.548046 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:30.547647 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.062328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.062377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.062485 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.062531 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:39.062519218 +0000 UTC m=+18.146783641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.062814 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:31.070386 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.062843 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:39.062834048 +0000 UTC m=+18.147098466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:31.163747 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.163621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:31.163925 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.163809 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:31.163925 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.163833 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:31.163925 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.163845 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:31.163925 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.163892 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:39.163878877 +0000 UTC m=+18.248143326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:31.404188 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.404155 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:34:31.409571 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.409536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:31.409719 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.409686 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:31.409778 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.409763 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:31.409852 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:31.409835 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:31.553188 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:31.553150 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" event={"ID":"814f2828-2c09-40ea-b910-02df6029fffd","Type":"ContainerStarted","Data":"55ebf013cc6cbe686761a8fb854e260085a08fe431afd24d5882be6c0896f511"} Oct 30 19:34:32.325121 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.324349 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:34:31.404179601Z","UUID":"b7809296-8736-4fc9-9a65-5eca1093dd28","Handler":null,"Name":"","Endpoint":""} Oct 30 19:34:32.326129 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.326099 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:34:32.326129 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.326136 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:34:32.407773 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.407536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:32.407929 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:32.407869 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:32.557797 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.557704 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="ffadf5a42a961b36ed3eb1670df703d731e038f948e798dd88cab318dd15f12d" exitCode=0 Oct 30 19:34:32.558217 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.557788 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"ffadf5a42a961b36ed3eb1670df703d731e038f948e798dd88cab318dd15f12d"} Oct 30 19:34:32.560667 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.560625 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" event={"ID":"814f2828-2c09-40ea-b910-02df6029fffd","Type":"ContainerStarted","Data":"97e8026c8ad61626f15abc1f9c78f57f73e663633da4f3bc0e269d09d6f3d2fc"} Oct 30 19:34:32.562976 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.562480 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t5d7p" event={"ID":"fad1332e-70fd-49ce-b1c2-48e50110f29b","Type":"ContainerStarted","Data":"041365ecad14362888cad18d69d291fe26d42fa6f217d402ff846627b0c99f2c"} Oct 30 19:34:32.564245 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.564216 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmsqs" event={"ID":"c58f03ba-4840-4237-88ee-82bf233ac315","Type":"ContainerStarted","Data":"d198e43260aaded5b7b9d1681cf43677fc370e63dd1794baaac51cf9d0753624"} Oct 30 19:34:32.608149 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.608103 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ftqg8" podStartSLOduration=6.116736016 podStartE2EDuration="11.608089427s" totalImagesPullingTime="5.491353411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:29.588209068 +0000 UTC m=+8.672473509" watchObservedRunningTime="2026-10-30 19:34:32.608089427 +0000 UTC m=+11.692353934" Oct 30 19:34:32.653297 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.653233 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t5d7p" podStartSLOduration=2.551503644 podStartE2EDuration="10.653213961s" totalImagesPullingTime="8.101710317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:32.652627957 +0000 UTC m=+11.736892397" watchObservedRunningTime="2026-10-30 19:34:32.653213961 +0000 UTC m=+11.737478401" Oct 30 19:34:32.707915 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.707850 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" podStartSLOduration=2.388785969 podStartE2EDuration="10.707830985s" totalImagesPullingTime="8.319045016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:32.707398766 +0000 UTC m=+11.791663204" watchObservedRunningTime="2026-10-30 19:34:32.707830985 +0000 UTC m=+11.792095427" Oct 30 19:34:32.765990 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:32.765925 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmsqs" podStartSLOduration=2.839104128 podStartE2EDuration="10.765904903s" totalImagesPullingTime="7.926800775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:32.765734856 +0000 UTC m=+11.849999297" watchObservedRunningTime="2026-10-30 19:34:32.765904903 +0000 UTC m=+11.850169343" Oct 30 19:34:33.408400 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.408305 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:33.408566 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:33.408454 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:33.408566 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.408506 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:33.408673 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:33.408607 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:33.571482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.571442 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twm5d" event={"ID":"557d0d40-9bfa-4d88-9ee2-ac07635e7daa","Type":"ContainerStarted","Data":"cda1693eca7eee30f83475754b4c93121e6a3ec664c24a8efcda0c19e9200a22"} Oct 30 19:34:33.574547 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.574511 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="27e75cb877c2a45f92ca922ce87276cfa996ec143a66037113e66bff1f01f430" exitCode=0 Oct 30 19:34:33.574959 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.574912 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"27e75cb877c2a45f92ca922ce87276cfa996ec143a66037113e66bff1f01f430"} Oct 30 19:34:33.615938 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:33.615864 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-twm5d" podStartSLOduration=4.407408564 podStartE2EDuration="12.615843613s" totalImagesPullingTime="8.208435049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:33.61526787 +0000 UTC m=+12.699532310" watchObservedRunningTime="2026-10-30 19:34:33.615843613 +0000 UTC m=+12.700108053" Oct 30 19:34:34.407599 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:34.407568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:34.407744 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:34.407691 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:34.579311 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:34.579275 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="fd3bee334fb0a5ec3139f2b80b36acd3bc456f90c8053c1591a7cc5ef9b1de9d" exitCode=0 Oct 30 19:34:34.580106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:34.579385 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"fd3bee334fb0a5ec3139f2b80b36acd3bc456f90c8053c1591a7cc5ef9b1de9d"} Oct 30 19:34:34.581890 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:34.581865 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" event={"ID":"4009f08d-95f6-4e73-ba56-2740e4e55336","Type":"ContainerStarted","Data":"799830fa803ed73d235ef83a25dbe6e8e8577b0e454eda83817e61b4eaf60edc"} Oct 30 19:34:35.407954 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:35.407915 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:35.407954 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:35.407965 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:35.408192 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:35.408082 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:35.408247 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:35.408204 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:36.407720 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:36.407685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:36.408347 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:36.407824 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:36.590696 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:36.590449 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerStarted","Data":"294e5ad1b0bce64e1f24e4d9cdacaf1b4ba6265140cc31254438e8270ebbe41c"} Oct 30 19:34:36.988509 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:36.988386 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:36.988678 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:36.988534 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:36.989051 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:36.989026 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:34:37.029844 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.029790 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" podStartSLOduration=4.452472133 podStartE2EDuration="15.029770189s" totalImagesPullingTime="10.577298056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:34.672453806 +0000 UTC m=+13.756718247" watchObservedRunningTime="2026-10-30 19:34:37.029770189 +0000 UTC m=+16.114034691" Oct 30 19:34:37.074744 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.074705 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7lfb_bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:37.261761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.261686 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmsqs_c58f03ba-4840-4237-88ee-82bf233ac315/dns-node-resolver/0.log" Oct 30 19:34:37.407570 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.407537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:37.407726 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.407573 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:37.407726 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:37.407667 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:37.408120 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:37.407763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:37.595422 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.595323 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="294e5ad1b0bce64e1f24e4d9cdacaf1b4ba6265140cc31254438e8270ebbe41c" exitCode=0 Oct 30 19:34:37.595570 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:37.595408 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"294e5ad1b0bce64e1f24e4d9cdacaf1b4ba6265140cc31254438e8270ebbe41c"} Oct 30 19:34:38.407228 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.407202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:38.407356 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:38.407326 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:38.460045 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.459876 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftqg8_3ac523a2-773b-493f-bb76-8faa6f8c3bb1/node-ca/0.log" Oct 30 19:34:38.601421 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.601390 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="82bce8838ab7fcbd61a8f4c171aa37b82d4aaf55bb44defcfd29e5d683673a0d" exitCode=0 Oct 30 19:34:38.601585 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.601480 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerDied","Data":"82bce8838ab7fcbd61a8f4c171aa37b82d4aaf55bb44defcfd29e5d683673a0d"} Oct 30 19:34:38.604705 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.604683 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:34:38.605054 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605032 2572 generic.go:356] "Generic (PLEG): container finished" podID="908aaccc-1937-4ac9-8614-28365a9fb984" containerID="ca8145735c42cab33b0aac9fa9c25a1886bf7c009215a35b0266afb42dfbacfe" exitCode=1 Oct 30 19:34:38.605130 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605079 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"276f23dffcd159c6b31ad6d1571b24f5ec77404dd9d9716fa061843e31818d58"} Oct 30 19:34:38.605168 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605138 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"a86c546f0a20910716953f81a7e788bfe8ffc2818d9dfe43bc0f9d82fb1d48f3"} Oct 30 19:34:38.605197 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605168 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"c988e53e8cc2af4e0cd8b7004798962c7038eb63c4da835517bcf9b8566d0c95"} Oct 30 19:34:38.605232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605197 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"c78f67d07ae55e696ea3b15fd155e0cec64311f89b2f5ccc4b91c5617057d7d0"} Oct 30 19:34:38.605265 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605230 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerDied","Data":"ca8145735c42cab33b0aac9fa9c25a1886bf7c009215a35b0266afb42dfbacfe"} Oct 30 19:34:38.605265 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:38.605259 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"c0735075ece307a65cf5ed2f16a7a291e249967f5c38b4c68def0ec917b3bba5"} Oct 30 19:34:39.126577 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.126492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:39.126577 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.126529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:39.126757 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.126638 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:39.126757 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.126692 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:55.126679596 +0000 UTC m=+34.210944014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:39.126757 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.126638 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:39.126852 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.126776 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:55.126762945 +0000 UTC m=+34.211027377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:39.227228 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.227187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:39.227426 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.227348 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:39.227426 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.227379 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:39.227426 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.227390 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:39.227513 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.227440 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:34:55.227426698 +0000 UTC m=+34.311691116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:39.408317 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.408238 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:39.408470 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.408354 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:39.408470 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.408411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:39.408561 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:39.408525 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:39.610793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:39.610763 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wksc9" event={"ID":"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488","Type":"ContainerStarted","Data":"3a2ca456ba600ad4184f19e5c5dd65c995282083df317f7fbb3c983c5986132d"} Oct 30 19:34:40.407884 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:40.407841 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:40.408077 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:40.407981 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:41.408516 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:41.408475 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:41.409140 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:41.408613 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:41.409140 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:41.408660 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:41.409140 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:41.408726 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:41.617492 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:41.617465 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:34:41.617856 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:41.617831 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"1ad55b4c499fc1310f445e772685b97a1b76a532a5c74e34a0e2c3fcb108fd79"} Oct 30 19:34:42.407536 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:42.407495 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:42.407682 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:42.407610 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:43.408169 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.408127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:43.408735 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:43.408262 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:43.408735 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.408309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:43.408735 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:43.408415 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:43.625879 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.625851 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:34:43.626241 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.626221 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"1b150e30479260e0b10c5d893ee3c90ac33d9bb49ba879d2a7b765f4a1ceeb4d"} Oct 30 19:34:43.626511 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.626481 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:43.626640 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.626624 2572 scope.go:122] "RemoveContainer" containerID="ca8145735c42cab33b0aac9fa9c25a1886bf7c009215a35b0266afb42dfbacfe" Oct 30 19:34:43.643471 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.643446 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:43.722033 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:43.721972 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wksc9" podStartSLOduration=11.291028219 podStartE2EDuration="21.721957395s" totalImagesPullingTime="10.430929176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:39.685587182 +0000 UTC m=+18.769851620" watchObservedRunningTime="2026-10-30 19:34:43.721957395 +0000 UTC m=+22.806221836" Oct 30 19:34:44.407838 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.407800 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:44.408009 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:44.407934 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:44.632214 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.632190 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:34:44.632639 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.632563 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" event={"ID":"908aaccc-1937-4ac9-8614-28365a9fb984","Type":"ContainerStarted","Data":"5c47a9bfc45ecf8e0789d166715bdda0cfc5168ebf260a18160a43d1481c6b92"} Oct 30 19:34:44.632896 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.632872 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:44.633025 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.632952 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:44.647910 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.647881 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:34:44.693466 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:44.693343 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" podStartSLOduration=9.174388057 podStartE2EDuration="23.693330743s" totalImagesPullingTime="14.518942686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:44.692643927 +0000 UTC m=+23.776908368" watchObservedRunningTime="2026-10-30 19:34:44.693330743 +0000 UTC m=+23.777595183" Oct 30 19:34:45.408106 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:45.408062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:45.408397 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:45.408231 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:45.408397 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:45.408262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:45.408397 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:45.408345 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:46.407919 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:46.407883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:46.408305 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:46.407996 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:47.408051 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:47.408005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:47.408051 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:47.408056 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:47.408517 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:47.408137 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:47.408517 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:47.408243 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:48.407753 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:48.407710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:48.408021 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:48.407826 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:49.408072 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:49.408032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:49.408072 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:49.408069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:49.408519 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:49.408141 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:49.408519 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:49.408270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:50.407245 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:50.407159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:50.407424 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:50.407275 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:51.408529 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:51.408492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:51.408988 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:51.408578 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:51.408988 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:51.408664 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:51.408988 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:51.408771 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:52.407513 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:52.407480 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:52.407693 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:52.407589 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:53.408130 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:53.408088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:53.408565 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:53.408223 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:53.408565 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:53.408267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:53.408565 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:53.408357 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:54.407801 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:54.407760 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:54.407988 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:54.407896 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:55.141558 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.141521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:55.141558 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.141558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:55.142045 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.141652 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:55.142045 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.141659 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:55.142045 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.141701 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret podName:92a56f5b-251a-4f17-9dc8-ed6434db1328 nodeName:}" failed. No retries permitted until 2026-10-30 19:35:27.141688466 +0000 UTC m=+66.225952886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret") pod "global-pull-secret-syncer-zzgh7" (UID: "92a56f5b-251a-4f17-9dc8-ed6434db1328") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:55.142045 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.141729 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs podName:a0797257-801e-4dac-b7ea-ec05f4a382f3 nodeName:}" failed. No retries permitted until 2026-10-30 19:35:27.141708841 +0000 UTC m=+66.225973276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs") pod "network-metrics-daemon-8lk6r" (UID: "a0797257-801e-4dac-b7ea-ec05f4a382f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:55.242297 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.242259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:55.242484 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.242445 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:55.242484 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.242471 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:55.242484 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.242481 2572 projected.go:196] Error preparing data for projected volume kube-api-access-9vqfj for pod openshift-network-diagnostics/network-check-target-xspfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:55.242579 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.242537 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj podName:6f0c8ec3-e700-43b5-a948-ded642dc21fe nodeName:}" failed. No retries permitted until 2026-10-30 19:35:27.242519321 +0000 UTC m=+66.326783741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vqfj" (UniqueName: "kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj") pod "network-check-target-xspfd" (UID: "6f0c8ec3-e700-43b5-a948-ded642dc21fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:55.407411 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.407308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:55.407578 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.407448 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:55.407578 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.407508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:55.407684 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.407629 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:55.830830 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.830784 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzgh7"] Oct 30 19:34:55.831010 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.830913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:55.831071 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.831027 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:55.834863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.834827 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xspfd"] Oct 30 19:34:55.835128 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.835105 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lk6r"] Oct 30 19:34:55.835256 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.835209 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:55.835256 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:55.835215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:55.835392 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.835331 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:55.835520 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:55.835492 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:57.408337 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:57.408150 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:57.408796 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:57.408157 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:57.408796 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:57.408436 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:57.408796 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:57.408202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:57.408796 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:57.408554 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:57.408796 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:57.408684 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:34:59.407813 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:59.407768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:34:59.407813 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:59.407791 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:34:59.407813 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:34:59.407813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:34:59.408300 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:59.407934 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xspfd" podUID="6f0c8ec3-e700-43b5-a948-ded642dc21fe" Oct 30 19:34:59.408300 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:59.408060 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lk6r" podUID="a0797257-801e-4dac-b7ea-ec05f4a382f3" Oct 30 19:34:59.408300 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:34:59.408144 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzgh7" podUID="92a56f5b-251a-4f17-9dc8-ed6434db1328" Oct 30 19:35:00.241014 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.240985 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeReady" Oct 30 19:35:00.241166 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.241110 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:35:00.335404 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.335351 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ts96r"] Oct 30 19:35:00.365284 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.365252 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pgmdj"] Oct 30 19:35:00.365452 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.365430 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.381277 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.381249 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts96r"] Oct 30 19:35:00.381430 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.381293 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgmdj"] Oct 30 19:35:00.381430 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.381391 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.384704 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.384631 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:35:00.384704 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.384666 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:35:00.384704 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.384686 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:35:00.384904 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.384711 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:35:00.384904 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.384811 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\"" Oct 30 19:35:00.397786 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.397754 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:35:00.402903 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.402882 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:35:00.403001 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.402977 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\"" Oct 30 19:35:00.443755 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.443719 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p24hl"] Oct 30 19:35:00.455738 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.455711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.468920 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.468887 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p24hl"] Oct 30 19:35:00.477723 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.477723 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.477915 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.477915 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.477915 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.478002 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrf6\" (UniqueName: \"kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.478002 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.477976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.478104 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.478059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.478104 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.478077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72j9m\" (UniqueName: \"kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.485332 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.485273 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:35:00.485519 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.485442 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:35:00.485519 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.485498 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\"" Oct 30 19:35:00.485609 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.485559 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:35:00.578761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.578761 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.579003 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578784 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb29l\" (UniqueName: \"kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.579003 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.579003 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578920 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579003 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578943 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrf6\" (UniqueName: \"kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.579003 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.578966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579132 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.579232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579183 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579390 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72j9m\" (UniqueName: \"kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579390 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.579390 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579524 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579615 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579597 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.579656 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.579631 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.583071 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.583043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.583189 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.583078 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.600824 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.600787 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrf6\" (UniqueName: \"kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.600967 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.600836 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72j9m\" (UniqueName: \"kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.680470 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.680390 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.680470 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.680431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb29l\" (UniqueName: \"kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.682796 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.682773 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.702496 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.702465 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb29l\" (UniqueName: \"kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.724496 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.724464 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:00.752837 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.752803 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:00.780651 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.780614 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:00.990408 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:00.990353 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:35:01.023181 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.023146 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:35:01.023340 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.023261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.040079 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040049 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:35:01.040260 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040127 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:35:01.040328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040290 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:35:01.040328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040292 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:35:01.040328 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040299 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:35:01.040482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040349 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:35:01.040482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040423 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:35:01.040482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.040424 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g55w6\"" Oct 30 19:35:01.114752 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.114723 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p24hl"] Oct 30 19:35:01.118515 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.118487 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgmdj"] Oct 30 19:35:01.119895 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.119870 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts96r"] Oct 30 19:35:01.120554 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:01.120513 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c38cf2_9e21_4d7f_836b_9983359012fc.slice/crio-7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215 WatchSource:0}: Error finding container 7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215: Status 404 returned error can't find the container with id 7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215 Oct 30 19:35:01.121423 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:01.121382 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode300e576_627d_46ea_a62d_567ec3f8d5c3.slice/crio-36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4 WatchSource:0}: Error finding container 36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4: Status 404 returned error can't find the container with id 36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4 Oct 30 19:35:01.184480 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184838 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.184838 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.184733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285296 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285630 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285630 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285630 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.285850 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.285826 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.286156 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.286082 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.286156 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.286083 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.286156 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.286083 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.288628 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.288600 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.288742 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.288601 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.317198 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.317163 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.340203 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.340170 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:01.411571 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.411544 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:01.411739 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.411670 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:01.411786 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.411743 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:01.430275 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430252 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\"" Oct 30 19:35:01.431059 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430652 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:35:01.431158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430673 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:35:01.431158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430709 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\"" Oct 30 19:35:01.431158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430764 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:35:01.431158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.430886 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:35:01.431158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.431014 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\"" Oct 30 19:35:01.671002 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.670905 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p24hl" event={"ID":"140004b8-1074-4ef5-a31e-9c9f52c46d8a","Type":"ContainerStarted","Data":"0e853cd61938490633f3ea97147c498a1005f06208457b7ac3d662b03e034a49"} Oct 30 19:35:01.672263 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.672241 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgmdj" event={"ID":"98c38cf2-9e21-4d7f-836b-9983359012fc","Type":"ContainerStarted","Data":"7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215"} Oct 30 19:35:01.673675 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.673655 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts96r" event={"ID":"e300e576-627d-46ea-a62d-567ec3f8d5c3","Type":"ContainerStarted","Data":"b84dc7ee84b77e117c568ed3970d4de0721a4b93960707b131bbb29d5f233333"} Oct 30 19:35:01.673883 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.673741 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts96r" event={"ID":"e300e576-627d-46ea-a62d-567ec3f8d5c3","Type":"ContainerStarted","Data":"36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4"} Oct 30 19:35:01.677127 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:01.677104 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:35:01.679971 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:01.679936 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4450c8f2_6d00_4a2d_aee2_4acd31855fea.slice/crio-74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa WatchSource:0}: Error finding container 74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa: Status 404 returned error can't find the container with id 74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa Oct 30 19:35:02.679821 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:02.679694 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts96r" event={"ID":"e300e576-627d-46ea-a62d-567ec3f8d5c3","Type":"ContainerStarted","Data":"f859f9ff7e67d29db04c61374416034cd8099666e3e7da8082179fba23604676"} Oct 30 19:35:02.681454 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:02.681409 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574f7c6c-ksbt6" event={"ID":"4450c8f2-6d00-4a2d-aee2-4acd31855fea","Type":"ContainerStarted","Data":"74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa"} Oct 30 19:35:03.686599 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:03.686517 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgmdj" event={"ID":"98c38cf2-9e21-4d7f-836b-9983359012fc","Type":"ContainerStarted","Data":"98395f6e1a04914089bbb2cfe52a3e0b03765e844e80152341e2a14ee4be9fe8"} Oct 30 19:35:04.130777 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.129382 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j"] Oct 30 19:35:04.145624 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.145590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.162547 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.160923 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j"] Oct 30 19:35:04.173584 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.173550 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:35:04.173584 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.173550 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:35:04.179693 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.179658 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:35:04.179871 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.179727 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:35:04.179940 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.179914 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gsp7d\"" Oct 30 19:35:04.180066 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.180047 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:35:04.196662 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.196594 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-csl8b"] Oct 30 19:35:04.209209 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209172 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks"] Oct 30 19:35:04.209387 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209263 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.209387 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88zq\" (UniqueName: \"kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.209520 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.209520 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.209607 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.209544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.226166 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.226134 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks"] Oct 30 19:35:04.226387 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.226287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.243256 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.243219 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:35:04.266715 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266632 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:35:04.266715 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266663 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:35:04.266715 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266669 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:35:04.267005 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266741 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zprrl\"" Oct 30 19:35:04.267005 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266847 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dpldx\"" Oct 30 19:35:04.267005 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266856 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:35:04.267005 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.266946 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:35:04.311022 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.310984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311208 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311208 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.311208 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311208 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311434 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311271 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.311434 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311434 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311434 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88zq\" (UniqueName: \"kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.311604 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311541 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n857\" (UniqueName: \"kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311650 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.311693 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.311734 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.311734 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.311824 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.311824 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h987g\" (UniqueName: \"kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.311919 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.311985 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.311962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.312043 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.312012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.312646 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.312625 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.316110 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.316082 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.316247 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.316109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.337065 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.337035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88zq\" (UniqueName: \"kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.412607 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412607 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412607 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412650 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.412863 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412889 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n857\" (UniqueName: \"kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.412969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413007 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h987g\" (UniqueName: \"kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413157 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.413616 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413328 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413616 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.413868 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.413827 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.414169 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.414147 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.414310 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.414284 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.415629 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.415603 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.415744 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.415626 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.415800 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.415783 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.417071 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.417049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.435534 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.435505 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n857\" (UniqueName: \"kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.435914 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.435891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h987g\" (UniqueName: \"kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.496541 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.496513 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:04.563566 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.562638 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:04.574329 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:04.574290 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ecb26d_be82_4455_8dae_e0b7489350ee.slice/crio-106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71 WatchSource:0}: Error finding container 106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71: Status 404 returned error can't find the container with id 106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71 Oct 30 19:35:04.600110 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.598847 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:04.692236 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.692044 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgmdj" event={"ID":"98c38cf2-9e21-4d7f-836b-9983359012fc","Type":"ContainerStarted","Data":"79ccd950235d188bbd56c01b157b88cf19ab159138b34602323ce0e608e040fa"} Oct 30 19:35:04.693035 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.692420 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:04.696502 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.695956 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts96r" event={"ID":"e300e576-627d-46ea-a62d-567ec3f8d5c3","Type":"ContainerStarted","Data":"aa7c2928f17d8282137dc8278b296e3e96563fd64371f83183b423ee429d47a1"} Oct 30 19:35:04.700775 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.700735 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p24hl" event={"ID":"140004b8-1074-4ef5-a31e-9c9f52c46d8a","Type":"ContainerStarted","Data":"507b6b4754ba86fc86b44e0a837da163f5310c6539fb31522a2baca0452598a3"} Oct 30 19:35:04.702444 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.702417 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csl8b" event={"ID":"d5ecb26d-be82-4455-8dae-e0b7489350ee","Type":"ContainerStarted","Data":"106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71"} Oct 30 19:35:04.738424 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.738358 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pgmdj" podStartSLOduration=2.425477693 podStartE2EDuration="4.738339191s" totalImagesPullingTime="2.312861498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:04.737327313 +0000 UTC m=+43.821591775" watchObservedRunningTime="2026-10-30 19:35:04.738339191 +0000 UTC m=+43.822603635" Oct 30 19:35:04.782340 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.782271 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p24hl" podStartSLOduration=1.356767317 podStartE2EDuration="4.782249451s" totalImagesPullingTime="3.425482134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:04.781427244 +0000 UTC m=+43.865691711" watchObservedRunningTime="2026-10-30 19:35:04.782249451 +0000 UTC m=+43.866513890" Oct 30 19:35:04.823542 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:04.823505 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8b2a9b_294c_4563_8b7e_6c6fa3d621ba.slice/crio-e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632 WatchSource:0}: Error finding container e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632: Status 404 returned error can't find the container with id e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632 Oct 30 19:35:04.825247 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.825223 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j"] Oct 30 19:35:04.829221 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.829167 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ts96r" podStartSLOduration=1.728240853 podStartE2EDuration="4.829149626s" totalImagesPullingTime="3.100908773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:04.827560559 +0000 UTC m=+43.911825026" watchObservedRunningTime="2026-10-30 19:35:04.829149626 +0000 UTC m=+43.913414068" Oct 30 19:35:04.916650 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:04.916618 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks"] Oct 30 19:35:05.553814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.553783 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:05.558235 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.557890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.574458 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.574145 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:35:05.574458 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.574299 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:35:05.574458 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.574327 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:35:05.574798 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.574770 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:35:05.582606 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.582577 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:35:05.582741 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.582632 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:35:05.582741 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.582703 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:35:05.582741 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.582723 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:35:05.582855 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.582789 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p99nr\"" Oct 30 19:35:05.590020 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.589973 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:35:05.598927 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.598896 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:05.627536 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627672 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627672 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627628 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627751 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627751 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627738 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627810 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627839 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627803 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627877 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627910 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627955 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627940 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.627986 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.627975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.628018 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.628006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.628053 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.628040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.709100 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.709056 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" event={"ID":"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba","Type":"ContainerStarted","Data":"22371b0808da67b8aaab1d96896e9effc2cf8dbe2cc885a0222267766ad7c80d"} Oct 30 19:35:05.709587 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.709141 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" event={"ID":"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba","Type":"ContainerStarted","Data":"6be31fd4aaee9f06f26ea95d572986ba831a385812839edbb2ad3c8f01f80ec9"} Oct 30 19:35:05.709587 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.709173 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" event={"ID":"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba","Type":"ContainerStarted","Data":"e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632"} Oct 30 19:35:05.711035 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.711005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" event={"ID":"724b9880-4314-41ad-93f7-40eb026a0fb2","Type":"ContainerStarted","Data":"0e4d50044561fb983ca790bf58221691a691e4df0cc8ef55ceece2fb4a373ee4"} Oct 30 19:35:05.713427 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.713397 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574f7c6c-ksbt6" event={"ID":"4450c8f2-6d00-4a2d-aee2-4acd31855fea","Type":"ContainerStarted","Data":"665f4aed7c2266d46ec120cdbd27593e0f9ad4fa17c32438cfbc87a3aae700ac"} Oct 30 19:35:05.715773 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.715742 2572 generic.go:356] "Generic (PLEG): container finished" podID="d5ecb26d-be82-4455-8dae-e0b7489350ee" containerID="55ea5e63c1614417d53c543fe347dd74bd2da62b6501e4a531af871b90e807d7" exitCode=0 Oct 30 19:35:05.716865 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.716842 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csl8b" event={"ID":"d5ecb26d-be82-4455-8dae-e0b7489350ee","Type":"ContainerDied","Data":"55ea5e63c1614417d53c543fe347dd74bd2da62b6501e4a531af871b90e807d7"} Oct 30 19:35:05.729461 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729625 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729707 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729763 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729814 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729873 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.729957 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730016 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.729994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730080 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.730022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730080 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.730058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730178 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.730102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730227 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.730198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.730277 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.730247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.731118 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.731086 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.731938 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.731905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.732326 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.732302 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.735511 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.735433 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.749951 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.749801 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.749951 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.749872 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.750845 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.750771 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.751481 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.751435 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.759026 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.758022 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.759026 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.758246 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.759026 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.758975 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.759026 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.759004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.761214 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.761167 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:05.816947 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.813133 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7574f7c6c-ksbt6" podStartSLOduration=2.168078043 podStartE2EDuration="5.813110293s" totalImagesPullingTime="3.64503225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:05.760902258 +0000 UTC m=+44.845166698" watchObservedRunningTime="2026-10-30 19:35:05.813110293 +0000 UTC m=+44.897374735" Oct 30 19:35:05.888571 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:05.888539 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:06.259253 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.259084 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:06.268318 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:06.268285 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4395c20_e68a_4714_99e3_6992284251f6.slice/crio-c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945 WatchSource:0}: Error finding container c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945: Status 404 returned error can't find the container with id c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945 Oct 30 19:35:06.423179 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.423154 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-747576895b-zqwt6"] Oct 30 19:35:06.426198 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.426178 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.447948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.447916 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:35:06.447948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.447916 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:35:06.447948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.447921 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:35:06.448215 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.448100 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:35:06.462873 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.462830 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:35:06.463195 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.463174 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-87vg7bm5h2rog\"" Oct 30 19:35:06.489166 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.489051 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-747576895b-zqwt6"] Oct 30 19:35:06.535191 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535155 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535412 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535412 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535412 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535684 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8bf\" (UniqueName: \"kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.535734 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.535716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.546545 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.546513 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vmphl\"" Oct 30 19:35:06.637025 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.636937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637025 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.636977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637243 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637243 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637243 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637482 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8bf\" (UniqueName: \"kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637677 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.637677 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.637568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.638235 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.638173 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.638235 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.638174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.640736 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.640707 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.641270 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.641176 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.641270 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.641210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.641507 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.641490 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.641676 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.641657 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.642045 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.642023 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.660623 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.660587 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8bf\" (UniqueName: \"kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.722879 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.722845 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" event={"ID":"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba","Type":"ContainerStarted","Data":"d1920bb3de2605975b0b9e223b816e79209a3e93bab40cc5a4a8a5413f02525a"} Oct 30 19:35:06.726031 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.725996 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" event={"ID":"724b9880-4314-41ad-93f7-40eb026a0fb2","Type":"ContainerStarted","Data":"5cdb0d4d08c400c11da4b92d9f4350fc68de15ead26ec3a7d457bc273ca94b41"} Oct 30 19:35:06.726164 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.726063 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" event={"ID":"724b9880-4314-41ad-93f7-40eb026a0fb2","Type":"ContainerStarted","Data":"1fdfd61e1eff8760025d271cd94ba26d767413e14156f20c0ac5a2480f679f9d"} Oct 30 19:35:06.726164 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.726093 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" event={"ID":"724b9880-4314-41ad-93f7-40eb026a0fb2","Type":"ContainerStarted","Data":"c26a06db400bd1185c7f7c0d4679921468f12f2e650b9ec86e9da2583f1e4df3"} Oct 30 19:35:06.728568 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.728527 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csl8b" event={"ID":"d5ecb26d-be82-4455-8dae-e0b7489350ee","Type":"ContainerStarted","Data":"af3d990e5ac5b25ba733b36d5be756941b8ab52d464b2a72050f99f79dc1e86f"} Oct 30 19:35:06.728694 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.728592 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csl8b" event={"ID":"d5ecb26d-be82-4455-8dae-e0b7489350ee","Type":"ContainerStarted","Data":"b4e13e0bc939d69b0fe937fa4711f5405ad7dce1bc3611b0f10a95dee533af3b"} Oct 30 19:35:06.729921 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.729891 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945"} Oct 30 19:35:06.742907 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.742879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:06.768780 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.767501 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" podStartSLOduration=1.68945792 podStartE2EDuration="2.767483475s" totalImagesPullingTime="1.078025555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.766581256 +0000 UTC m=+45.850845727" watchObservedRunningTime="2026-10-30 19:35:06.767483475 +0000 UTC m=+45.851747918" Oct 30 19:35:06.821091 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.821040 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" podStartSLOduration=1.477171582 podStartE2EDuration="2.821026345s" totalImagesPullingTime="1.343854763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.820570825 +0000 UTC m=+45.904835267" watchObservedRunningTime="2026-10-30 19:35:06.821026345 +0000 UTC m=+45.905290785" Oct 30 19:35:06.873428 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:06.873349 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-csl8b" podStartSLOduration=1.868185543 podStartE2EDuration="2.873339047s" totalImagesPullingTime="923.602794ms" totalInitContainerRuntime="81.55071ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.872288837 +0000 UTC m=+45.956553289" watchObservedRunningTime="2026-10-30 19:35:06.873339047 +0000 UTC m=+45.957603487" Oct 30 19:35:07.054839 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:07.054808 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-747576895b-zqwt6"] Oct 30 19:35:07.056836 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:07.056802 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7977864_d9ee_43d9_9c44_6611a31dba60.slice/crio-f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9 WatchSource:0}: Error finding container f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9: Status 404 returned error can't find the container with id f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9 Oct 30 19:35:07.735193 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:07.735155 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9"} Oct 30 19:35:07.736909 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:07.736879 2572 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="b13258187328279b1f5d16c20293e4f174f3583fe53b66cbc8c729031fc78ee3" exitCode=0 Oct 30 19:35:07.737054 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:07.736995 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"b13258187328279b1f5d16c20293e4f174f3583fe53b66cbc8c729031fc78ee3"} Oct 30 19:35:08.765800 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.765760 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-695b5bb69b-wlqqm"] Oct 30 19:35:08.768303 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.768277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.785079 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.785051 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:35:08.785249 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.785237 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:35:08.799394 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.799327 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:35:08.799597 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.799411 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j6mrb\"" Oct 30 19:35:08.799597 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.799562 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fd02f8mhdkfnh\"" Oct 30 19:35:08.799597 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.799577 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:35:08.800984 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.800956 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-695b5bb69b-wlqqm"] Oct 30 19:35:08.860242 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860441 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860441 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860441 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wvp\" (UniqueName: \"kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860441 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860722 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860452 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.860722 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.860519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.961139 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961100 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.961323 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wvp\" (UniqueName: \"kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.961323 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.961323 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.961323 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.962151 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961598 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.962151 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.962151 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.961695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.962151 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.962011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.962664 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.962314 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.965639 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.965196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.965799 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.965754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.965983 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.965959 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.986265 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.986235 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wvp\" (UniqueName: \"kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:08.993498 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.993404 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z"] Oct 30 19:35:08.996612 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:08.996589 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.028966 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.028935 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z"] Oct 30 19:35:09.029947 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.029925 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bgfdp\"" Oct 30 19:35:09.029947 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.029944 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:35:09.062594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.062348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.062594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.062493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kqf\" (UniqueName: \"kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.083397 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.083356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:09.173832 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.163839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.173832 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.163960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kqf\" (UniqueName: \"kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.173832 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.172802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.192455 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.192423 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kqf\" (UniqueName: \"kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.321884 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.321863 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:09.434101 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.434067 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-695b5bb69b-wlqqm"] Oct 30 19:35:09.518087 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.518059 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84748fcdfb-kqjdr"] Oct 30 19:35:09.520663 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.520643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.540655 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540622 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 19:35:09.540655 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540641 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 19:35:09.540655 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540621 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 19:35:09.540973 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540685 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 19:35:09.540973 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540747 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 19:35:09.540973 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.540772 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fc58z\"" Oct 30 19:35:09.568435 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.568404 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 19:35:09.569884 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.569810 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84748fcdfb-kqjdr"] Oct 30 19:35:09.572489 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572489 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jtg\" (UniqueName: \"kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572664 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572676 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572843 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572899 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572950 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.572993 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.572963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.622035 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.621922 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z"] Oct 30 19:35:09.624729 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:09.624701 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4063bcb_d0a9_4bab_89a1_fe44f1a9e0e7.slice/crio-39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c WatchSource:0}: Error finding container 39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c: Status 404 returned error can't find the container with id 39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jtg\" (UniqueName: \"kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.673839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675004 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.674687 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.675577 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.675174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.676100 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.676069 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.677693 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.677476 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.677693 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.677652 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.677928 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.677886 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.678648 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.678625 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.697951 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.697921 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jtg\" (UniqueName: \"kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:09.745949 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.745910 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" event={"ID":"99ab51f7-f13e-41df-985a-172ac7fe4927","Type":"ContainerStarted","Data":"6ae60f5e92d9a48744aeb354011e951804b0360175a1edea404752b26956c8bd"} Oct 30 19:35:09.748230 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.748194 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"545a767f4afea5cd52e420eebae8f6c2541a072005cde50afa8da4c0ba9a0ae9"} Oct 30 19:35:09.748377 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.748264 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"30d9adf0482fbed950bb4f686a3fdc1888021f34bfa845384f181979f87f3827"} Oct 30 19:35:09.748377 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.748305 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"1209d97fdd438e66187349b5be0e7c8df465c6e6434760184b29196955cac402"} Oct 30 19:35:09.749494 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.749471 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" event={"ID":"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7","Type":"ContainerStarted","Data":"39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c"} Oct 30 19:35:09.754816 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.754789 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"b8d898edb5f54bdc1ad25f2217bfda9bfbe69c766dcce8818c758cb593cedff4"} Oct 30 19:35:09.754944 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.754845 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"5820904f4fa174260a2a17648db8dc8c373a39c062f8ae9aa11af6b10d054d7c"} Oct 30 19:35:09.754944 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.754872 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"c2ca2eb7fdd7c0c706aa97c787e38e379d090390fd3434c66e056e4531846a7b"} Oct 30 19:35:09.754944 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.754900 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"67cd90bd5cb78bdd7bca02acdb70ec730434ffcd8ed41a4e8d5cbf15a1fb90a2"} Oct 30 19:35:09.754944 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.754922 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"088c6cca94acbbddf2e0a0e75e8cc8298bff2b28864dddb7f9dadda0e44e3cca"} Oct 30 19:35:09.837455 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:09.837342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:10.172392 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.172348 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84748fcdfb-kqjdr"] Oct 30 19:35:10.185508 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:10.185474 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc76c3f5_d680_4468_a8ae_b100e41e253c.slice/crio-868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38 WatchSource:0}: Error finding container 868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38: Status 404 returned error can't find the container with id 868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38 Oct 30 19:35:10.772738 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.772683 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"48117e282e8ed96bd31de690708ee3cf001face7ef663933cfe42baf4f085027"} Oct 30 19:35:10.773053 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.773029 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:10.773179 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.773091 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"9d185a7bc0053952bc342adc4cb5960338830185193acfd220e4bbe8531c4fd4"} Oct 30 19:35:10.773179 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.773134 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" event={"ID":"e7977864-d9ee-43d9-9c44-6611a31dba60","Type":"ContainerStarted","Data":"39913bd10c39d76140aea6bbfb40ec4d175f68b4d05a57b24887bc59972736f4"} Oct 30 19:35:10.774394 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.774343 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" event={"ID":"dc76c3f5-d680-4468-a8ae-b100e41e253c","Type":"ContainerStarted","Data":"868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38"} Oct 30 19:35:10.778809 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.778779 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerStarted","Data":"412a588f6109afad7e79e4c6ff2be232f441e0635e1e9c30b3da3317a7fdbbcd"} Oct 30 19:35:10.827657 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:10.827596 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" podStartSLOduration=2.243871273 podStartE2EDuration="4.82757543s" totalImagesPullingTime="2.583704157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:10.826719693 +0000 UTC m=+49.910984146" watchObservedRunningTime="2026-10-30 19:35:10.82757543 +0000 UTC m=+49.911839872" Oct 30 19:35:11.101675 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.101558 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.300138662 podStartE2EDuration="6.10153613s" totalImagesPullingTime="2.78653332s" totalInitContainerRuntime="14.864148ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:10.888979207 +0000 UTC m=+49.973243650" watchObservedRunningTime="2026-10-30 19:35:11.10153613 +0000 UTC m=+50.185800572" Oct 30 19:35:11.102734 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.102704 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:11.108411 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.108355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.126240 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126211 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:35:11.126476 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126260 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:35:11.126546 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126510 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:35:11.126716 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126695 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:35:11.126909 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126777 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:35:11.126909 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126848 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:35:11.127028 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126923 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:35:11.127028 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.126953 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:35:11.127249 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127234 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:35:11.127436 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127416 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:35:11.127567 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127530 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9e0lqdo2eio9\"" Oct 30 19:35:11.127638 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127587 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:35:11.127684 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127653 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgcwn\"" Oct 30 19:35:11.127899 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.127731 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:35:11.143306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.143263 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:11.143823 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.143801 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:35:11.143941 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.143851 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:35:11.193158 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193308 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193308 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193473 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193498 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193576 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193594 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.193785 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.194047 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.194047 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.194047 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.193966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.194047 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.194016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294548 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294519 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294548 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294683 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.294793 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.294988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.295010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295185 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.295045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.295488 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.295457 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.297194 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.296131 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.297194 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.296688 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.298010 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.297985 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.298806 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.298008 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.298806 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.298051 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.298806 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.298255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.298806 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.298473 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.299784 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.299425 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.299784 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.299498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.299948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.299782 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.299948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.299827 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.299948 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.299862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.300353 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.300332 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.300527 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.300508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.300527 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.300506 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.300813 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.300791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.301232 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.301206 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.301434 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.301411 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.301548 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.301514 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.302305 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.302277 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.302674 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.302650 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.302750 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.302725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.316388 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.316337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.340323 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.340279 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:11.340517 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.340389 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:11.348757 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.348730 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:11.435437 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.435341 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:11.772820 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.772786 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:11.785543 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:11.785511 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5533697_e0c7_42b4_958f_cbfc3541bc3e.slice/crio-fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c WatchSource:0}: Error finding container fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c: Status 404 returned error can't find the container with id fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c Oct 30 19:35:11.786079 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.785971 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" event={"ID":"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7","Type":"ContainerStarted","Data":"b13a4c0dd46fb310a2bc246a2d7e6c7340b0d9a292244170dbcc6eb785733a31"} Oct 30 19:35:11.786079 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.786044 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:11.789135 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.789100 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" event={"ID":"99ab51f7-f13e-41df-985a-172ac7fe4927","Type":"ContainerStarted","Data":"e8a8bc3d3825b142f7b5d91b1743710402a27a0e1efc1526f7c3fb2ca7999e3b"} Oct 30 19:35:11.792253 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.792230 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:11.794860 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.794841 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:11.829247 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.829199 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" podStartSLOduration=2.276502688 podStartE2EDuration="3.829182951s" totalImagesPullingTime="1.552680263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:11.828004561 +0000 UTC m=+50.912269023" watchObservedRunningTime="2026-10-30 19:35:11.829182951 +0000 UTC m=+50.913447392" Oct 30 19:35:11.916907 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:11.916848 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" podStartSLOduration=2.566041754 podStartE2EDuration="3.91682948s" totalImagesPullingTime="1.350787726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:11.915238728 +0000 UTC m=+50.999503162" watchObservedRunningTime="2026-10-30 19:35:11.91682948 +0000 UTC m=+51.001093922" Oct 30 19:35:12.793820 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.793781 2572 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" exitCode=0 Oct 30 19:35:12.794279 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.793866 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} Oct 30 19:35:12.794279 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.793953 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c"} Oct 30 19:35:12.796277 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.796249 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" event={"ID":"dc76c3f5-d680-4468-a8ae-b100e41e253c","Type":"ContainerStarted","Data":"a0eb0e11b5c9b9fea2b2903aee93157ac281761ba4877989f69e6706647a29e8"} Oct 30 19:35:12.796393 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.796303 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" event={"ID":"dc76c3f5-d680-4468-a8ae-b100e41e253c","Type":"ContainerStarted","Data":"8d5a892c1a7f8b83a628b8532012c6c83eb2dce82fcac1c54d1be09a3f235d15"} Oct 30 19:35:12.796393 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.796324 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" event={"ID":"dc76c3f5-d680-4468-a8ae-b100e41e253c","Type":"ContainerStarted","Data":"d37720aff8adc2503fc60e558fc3c3693102bc83875fd6c7179a7a6a2cd2490a"} Oct 30 19:35:12.897657 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:12.897596 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" podStartSLOduration=2.086197912 podStartE2EDuration="3.897581262s" totalImagesPullingTime="1.81138335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.896111858 +0000 UTC m=+51.980376307" watchObservedRunningTime="2026-10-30 19:35:12.897581262 +0000 UTC m=+51.981845700" Oct 30 19:35:14.720021 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:14.719985 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:15.819935 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:15.819887 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} Oct 30 19:35:15.820306 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:15.819984 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} Oct 30 19:35:16.647830 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.647804 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:16.797138 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.797111 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:16.827331 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.827288 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} Oct 30 19:35:16.827833 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.827354 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} Oct 30 19:35:16.827833 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.827399 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} Oct 30 19:35:16.827833 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.827423 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerStarted","Data":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} Oct 30 19:35:16.910993 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:16.910898 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.058895639 podStartE2EDuration="5.91088255s" totalImagesPullingTime="2.830629178s" totalInitContainerRuntime="21.357733ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.909710755 +0000 UTC m=+55.993975196" watchObservedRunningTime="2026-10-30 19:35:16.91088255 +0000 UTC m=+55.995146991" Oct 30 19:35:21.435472 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:21.435432 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:27.161795 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.161757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:27.162213 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.161827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:27.190976 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.190940 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:35:27.191169 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.190992 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:35:27.194802 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.194765 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:27.194990 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.194765 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:27.254522 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.254493 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\"" Oct 30 19:35:27.259576 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.259546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:27.263116 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.263091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:27.278484 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.278461 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:35:27.298958 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.298924 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:35:27.303339 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.303312 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\"" Oct 30 19:35:27.306958 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.306927 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:27.308424 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.308406 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:27.558710 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.558679 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzgh7"] Oct 30 19:35:27.559743 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:27.559711 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a56f5b_251a_4f17_9dc8_ed6434db1328.slice/crio-ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492 WatchSource:0}: Error finding container ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492: Status 404 returned error can't find the container with id ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492 Oct 30 19:35:27.576385 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.576329 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\"" Oct 30 19:35:27.581478 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.581448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:27.616188 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.616164 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lk6r"] Oct 30 19:35:27.618325 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:27.618296 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0797257_801e_4dac_b7ea_ec05f4a382f3.slice/crio-dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d WatchSource:0}: Error finding container dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d: Status 404 returned error can't find the container with id dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d Oct 30 19:35:27.873175 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.873081 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lk6r" event={"ID":"a0797257-801e-4dac-b7ea-ec05f4a382f3","Type":"ContainerStarted","Data":"dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d"} Oct 30 19:35:27.874265 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.874240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzgh7" event={"ID":"92a56f5b-251a-4f17-9dc8-ed6434db1328","Type":"ContainerStarted","Data":"ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492"} Oct 30 19:35:27.881577 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:27.881555 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xspfd"] Oct 30 19:35:27.882606 ip-10-0-132-4 kubenswrapper[2572]: W1030 19:35:27.882581 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0c8ec3_e700_43b5_a948_ded642dc21fe.slice/crio-997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb WatchSource:0}: Error finding container 997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb: Status 404 returned error can't find the container with id 997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb Oct 30 19:35:28.882889 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:28.882858 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lk6r" event={"ID":"a0797257-801e-4dac-b7ea-ec05f4a382f3","Type":"ContainerStarted","Data":"a9183e77439d0bc9fb708ab7afaec5d801893e026a9a257e9019a70b1a4aefaf"} Oct 30 19:35:28.884812 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:28.884759 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xspfd" event={"ID":"6f0c8ec3-e700-43b5-a948-ded642dc21fe","Type":"ContainerStarted","Data":"997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb"} Oct 30 19:35:29.083684 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:29.083646 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:29.084442 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:29.084422 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:29.891934 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:29.891892 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lk6r" event={"ID":"a0797257-801e-4dac-b7ea-ec05f4a382f3","Type":"ContainerStarted","Data":"d900118ce097fc70cbdc7bc0a3815367fef662ca6d5cdd104372e901032dbb7e"} Oct 30 19:35:29.941009 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:29.939947 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8lk6r" podStartSLOduration=66.89689407 podStartE2EDuration="1m7.939926497s" totalImagesPullingTime="1.043032427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:29.938533077 +0000 UTC m=+69.022797531" watchObservedRunningTime="2026-10-30 19:35:29.939926497 +0000 UTC m=+69.024190941" Oct 30 19:35:30.761683 ip-10-0-132-4 kubenswrapper[2572]: E1030 19:35:30.761598 2572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:35:31.902985 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:31.902949 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xspfd" event={"ID":"6f0c8ec3-e700-43b5-a948-ded642dc21fe","Type":"ContainerStarted","Data":"513e6581c5cd23036ac24d02fc13f466ca6e4aa6a7f17c703855519ab21b3ff8"} Oct 30 19:35:31.903525 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:31.903047 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:31.954977 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:31.954918 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xspfd" podStartSLOduration=67.810773229 podStartE2EDuration="1m10.954895601s" totalImagesPullingTime="3.144122372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:31.95281639 +0000 UTC m=+71.037080832" watchObservedRunningTime="2026-10-30 19:35:31.954895601 +0000 UTC m=+71.039160043" Oct 30 19:35:31.998198 ip-10-0-132-4 kubenswrapper[2572]: I1030 19:35:31.998167 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:35:32.477147 ip-10-0-132-4 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:35:32.499862 ip-10-0-132-4 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:35:32.500124 ip-10-0-132-4 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:35:32.500433 ip-10-0-132-4 systemd[1]: kubelet.service: Consumed 5.037s CPU time. Oct 30 19:35:32.517902 ip-10-0-132-4 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:35:32.582979 ip-10-0-132-4 kubenswrapper[5991]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:35:32.582979 ip-10-0-132-4 kubenswrapper[5991]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:35:32.582979 ip-10-0-132-4 kubenswrapper[5991]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:35:32.582979 ip-10-0-132-4 kubenswrapper[5991]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:35:32.585097 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585008 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.585097 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585096 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.585097 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585100 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585104 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585109 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585112 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585115 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585119 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585124 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585128 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585131 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585134 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585137 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585140 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585142 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585145 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585148 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585151 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585154 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585156 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585159 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.585200 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585162 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585164 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585171 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585174 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585177 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585193 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585197 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585199 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585202 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585205 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585208 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585210 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585213 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585217 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585221 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585224 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585227 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585230 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585233 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.585694 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585235 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585238 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585241 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585244 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585246 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585250 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585252 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585255 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585258 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585260 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585263 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585266 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585268 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585271 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585275 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585278 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585281 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585283 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585286 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.586219 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585294 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585297 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585299 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585302 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585304 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585307 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585310 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585312 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585315 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585318 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585320 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585323 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585325 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585328 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585330 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585333 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585336 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585338 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585342 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585345 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585347 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.586727 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585350 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585353 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585355 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585358 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585385 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585389 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585392 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585395 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585398 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585401 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585403 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585406 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585415 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585418 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585421 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585424 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585426 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585429 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585432 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.587248 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585436 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585439 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585442 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585445 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585447 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585450 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.585453 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586255 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586262 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586266 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586269 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586271 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586274 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586277 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586279 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586282 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586285 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586287 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586290 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586293 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.587751 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586296 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586299 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586302 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586305 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586308 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586310 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586321 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586323 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586326 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586329 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586332 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586335 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586337 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586340 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586343 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586346 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586348 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586351 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586353 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586356 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.588257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586358 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586381 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586387 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586390 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586393 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586396 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586398 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586401 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586404 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586406 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586409 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586411 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586415 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586419 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586421 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586424 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586426 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586429 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586432 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586439 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.588757 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586442 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586444 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586447 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586450 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586452 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586455 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586459 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586463 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586466 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586469 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586472 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586474 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586477 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586479 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586482 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586485 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586487 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586490 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586494 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.589260 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586496 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586499 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586502 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586505 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586508 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586511 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586513 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586516 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586518 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586521 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586524 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586526 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586528 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586535 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586538 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586540 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586543 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586546 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586548 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.590041 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586551 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586553 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586556 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586559 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586561 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586563 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586566 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586569 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586571 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586574 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586577 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586581 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586585 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586588 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.586591 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587163 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587168 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587170 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587173 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587176 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.590763 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587179 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587182 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587185 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587187 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587190 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587193 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587196 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587199 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587203 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587207 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587210 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587214 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587217 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587219 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587222 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587225 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587227 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587230 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587233 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.591275 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587235 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587238 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587240 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587243 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587246 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587249 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587252 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587255 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587258 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587260 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587263 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587266 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587268 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587271 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587273 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587277 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587290 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587294 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587297 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.591789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587301 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587310 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587313 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587316 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587318 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587322 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587324 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587327 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587330 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587332 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587335 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587338 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587340 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587346 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587349 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587351 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587354 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587358 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587375 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.592332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587378 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587381 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587383 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587386 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587389 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587391 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587394 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587397 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587399 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587402 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587405 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587407 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587410 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587412 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587415 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587424 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587427 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587429 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587432 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587435 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.592835 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587437 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587440 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587442 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587445 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587447 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587450 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587454 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587456 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587459 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587462 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587466 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587468 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587471 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587473 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587475 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587478 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587481 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587483 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587486 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587488 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.593321 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587491 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587493 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587496 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.587499 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587604 5991 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587624 5991 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587634 5991 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587638 5991 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587650 5991 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587654 5991 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587659 5991 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587668 5991 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587672 5991 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587675 5991 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587679 5991 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587682 5991 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587685 5991 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587688 5991 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587693 5991 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587696 5991 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587699 5991 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587702 5991 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587711 5991 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:35:32.593823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587714 5991 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587717 5991 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587720 5991 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587724 5991 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587728 5991 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587731 5991 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587734 5991 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587737 5991 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587740 5991 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587743 5991 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587746 5991 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587749 5991 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587753 5991 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587757 5991 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587761 5991 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587764 5991 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587767 5991 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587770 5991 flags.go:64] FLAG: --enable-server="true" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587779 5991 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587792 5991 flags.go:64] FLAG: --event-burst="100" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587795 5991 flags.go:64] FLAG: --event-qps="50" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587798 5991 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587801 5991 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587804 5991 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:35:32.594394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587809 5991 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587812 5991 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587815 5991 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587819 5991 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587822 5991 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587826 5991 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587829 5991 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587833 5991 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587836 5991 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587839 5991 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587842 5991 flags.go:64] FLAG: --feature-gates="" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587847 5991 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587850 5991 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587853 5991 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587856 5991 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587859 5991 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587862 5991 flags.go:64] FLAG: --help="false" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587865 5991 flags.go:64] FLAG: --hostname-override="ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587868 5991 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587871 5991 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587874 5991 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587878 5991 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587881 5991 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587884 5991 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:35:32.595015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587887 5991 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587890 5991 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587893 5991 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587902 5991 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587906 5991 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587909 5991 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587912 5991 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587915 5991 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587918 5991 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587921 5991 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587924 5991 flags.go:64] FLAG: --lock-file="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587929 5991 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587932 5991 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587935 5991 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587940 5991 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587945 5991 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587948 5991 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587951 5991 flags.go:64] FLAG: --logging-format="text" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587954 5991 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587957 5991 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587960 5991 flags.go:64] FLAG: --manifest-url="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587963 5991 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587968 5991 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587971 5991 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587975 5991 flags.go:64] FLAG: --max-pods="110" Oct 30 19:35:32.595712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587978 5991 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587981 5991 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587984 5991 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587987 5991 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587990 5991 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587993 5991 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.587996 5991 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588004 5991 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588007 5991 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588010 5991 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588014 5991 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588022 5991 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588025 5991 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588029 5991 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588032 5991 flags.go:64] FLAG: --port="10250" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588034 5991 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588038 5991 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08cb5a7224d0c2cf8" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588041 5991 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588045 5991 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588048 5991 flags.go:64] FLAG: --register-node="true" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588051 5991 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588055 5991 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588059 5991 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588062 5991 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588065 5991 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588069 5991 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:35:32.596318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588072 5991 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588075 5991 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588078 5991 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588081 5991 flags.go:64] FLAG: --runonce="false" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588084 5991 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588088 5991 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588091 5991 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588094 5991 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588097 5991 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588100 5991 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588103 5991 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588106 5991 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588109 5991 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588111 5991 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588114 5991 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588117 5991 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588120 5991 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588123 5991 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588132 5991 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588137 5991 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588140 5991 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588143 5991 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588150 5991 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588153 5991 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588157 5991 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:35:32.597098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588160 5991 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588163 5991 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588166 5991 flags.go:64] FLAG: --v="2" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588172 5991 flags.go:64] FLAG: --version="false" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588177 5991 flags.go:64] FLAG: --vmodule="" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588181 5991 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588184 5991 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588406 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588414 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588417 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588420 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588423 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588426 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588429 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588431 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588434 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588437 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588440 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588449 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588452 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588455 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.597737 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588457 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588460 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588462 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588465 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588468 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588476 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588478 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588481 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588484 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588487 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588491 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588493 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588496 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588498 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588502 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588504 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588507 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588509 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588512 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588514 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.598270 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588517 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588519 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588522 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588524 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588527 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588529 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588532 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588534 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588537 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588539 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588541 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588544 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588546 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588549 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588551 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588554 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588556 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588558 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588568 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588571 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.598796 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588573 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588576 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588581 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588584 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588588 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588591 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588594 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588596 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588599 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588603 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588606 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588609 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588612 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588615 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588617 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588620 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588622 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588625 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.599421 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588627 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588629 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588632 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588634 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588637 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588639 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588642 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588645 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588647 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588650 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588652 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588655 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588657 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588665 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588668 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588671 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588677 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588681 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588685 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.599920 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588689 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588693 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588697 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588700 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588702 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588705 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588708 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588710 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588713 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588716 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588718 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588721 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588723 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588726 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588728 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.588758 5991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:35:32.600474 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588805 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588811 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588814 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588816 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588819 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588822 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588824 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588827 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588829 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588832 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588834 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588844 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588847 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588851 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588854 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588856 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588859 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588861 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588864 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588867 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.600919 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588869 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588872 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588874 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588877 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588879 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588882 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588885 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588888 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588891 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588894 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588896 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588899 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588902 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588904 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588907 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588909 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588912 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588914 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588917 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588920 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.601585 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588923 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588925 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588928 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588930 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588941 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588946 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588949 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588951 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588954 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588956 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588959 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588961 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588964 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588967 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588969 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588972 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588974 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588977 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588979 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588983 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.602143 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588986 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588988 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588991 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588994 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588996 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.588999 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589001 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589004 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589007 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589009 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589011 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589014 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589017 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589019 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589022 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589024 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589027 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589036 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589039 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589042 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.602688 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589044 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589047 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589049 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589052 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589055 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589057 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589061 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589064 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589069 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589072 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589075 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589079 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589082 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589085 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589090 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589094 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589097 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589100 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.603274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589103 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589105 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589108 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589110 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589113 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589115 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589118 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.589120 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.593846 5991 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.593867 5991 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.593888 5991 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.593894 5991 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593970 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593976 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593980 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593983 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593986 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593989 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593992 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593995 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.603777 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.593998 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594000 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594003 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594006 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594009 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594011 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594015 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594018 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594021 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594024 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594026 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594029 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594032 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594034 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594037 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594041 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594044 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594046 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594049 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.604265 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594052 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594054 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594057 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594059 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594063 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594067 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594071 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594074 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594077 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594080 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594082 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594086 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594091 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594094 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594097 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594100 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594103 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594105 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594108 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.604803 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594112 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594115 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594118 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594120 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594123 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594126 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594128 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594131 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594133 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594136 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594139 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594142 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594144 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594147 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594149 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594152 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594155 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594158 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594160 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594163 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.605535 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594165 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594168 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594171 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594173 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594176 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594179 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594181 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594184 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594187 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594190 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594192 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594195 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594198 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594203 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594205 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594208 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594211 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594213 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594216 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594218 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.606268 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594221 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594224 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594227 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594229 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594232 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594234 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594237 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594240 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594242 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594245 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594248 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594250 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594253 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594255 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594258 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594260 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594263 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594266 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594269 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.606794 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594271 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.594298 5991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594491 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594498 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594501 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594505 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594508 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594511 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594514 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594517 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594520 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594522 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594525 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594528 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594530 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594533 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.607264 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594536 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594539 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594542 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594545 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594548 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594550 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594553 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594555 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594559 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594578 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594583 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594586 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594590 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594593 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594595 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594598 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594601 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594604 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594606 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594609 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.607760 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594612 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594616 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594621 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594625 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594631 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594634 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594637 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594640 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594642 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594645 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594648 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594650 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594653 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594655 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594659 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594663 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594666 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594669 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.608257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594671 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594674 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594676 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594679 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594682 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594684 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594687 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594689 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594692 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594697 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594700 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594702 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594705 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594708 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594711 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594713 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594716 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594719 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594721 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.608726 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594724 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594727 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594730 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594733 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594744 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594747 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594750 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594752 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594755 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594758 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594760 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594763 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594766 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594768 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594771 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594774 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594777 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594779 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594782 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594785 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.609227 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594787 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594790 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594794 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594797 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594800 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594802 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594805 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594808 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594811 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594813 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594816 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594819 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594821 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594824 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594827 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.594852 5991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:35:32.609834 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594886 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594890 5991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594894 5991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594897 5991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594899 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594902 5991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594904 5991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594907 5991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594910 5991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594913 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594915 5991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594918 5991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594920 5991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594923 5991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594926 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594928 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594931 5991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594933 5991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594936 5991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594940 5991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:35:32.610310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594943 5991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594947 5991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594951 5991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594955 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594959 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594962 5991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594964 5991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594967 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594970 5991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594972 5991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594975 5991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594978 5991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594980 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594983 5991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594988 5991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594991 5991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594994 5991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594997 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:35:32.610816 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.594999 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595002 5991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595005 5991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595007 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595010 5991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595012 5991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595015 5991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595018 5991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595021 5991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595024 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595027 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595030 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595032 5991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595036 5991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595039 5991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595042 5991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595044 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595047 5991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595050 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:35:32.611250 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595053 5991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595056 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595059 5991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595061 5991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595064 5991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595066 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595069 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595072 5991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595075 5991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595077 5991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595080 5991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595083 5991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595085 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595087 5991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595090 5991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595093 5991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595095 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595098 5991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595100 5991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595103 5991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:35:32.611774 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595107 5991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595110 5991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595112 5991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595115 5991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595118 5991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595120 5991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595124 5991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595127 5991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595129 5991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595132 5991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595134 5991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595137 5991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595139 5991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595142 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595145 5991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595148 5991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595150 5991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595153 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595156 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595159 5991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:35:32.612280 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595162 5991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595164 5991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595167 5991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595170 5991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595172 5991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595175 5991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595177 5991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595180 5991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:32.595184 5991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.595246 5991 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596276 5991 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596332 5991 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596531 5991 server.go:1014] "Starting client certificate rotation" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596681 5991 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596749 5991 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:29:21 +0000 UTC" deadline="2028-07-06 00:13:10.994363085 +0000 UTC" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596797 5991 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14740h37m38.397569463s" Oct 30 19:35:32.612792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.596995 5991 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:35:32.613196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.598122 5991 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:35:32.613196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.599620 5991 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:35:32.613196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.601585 5991 log.go:25] "Validated CRI v1 image API" Oct 30 19:35:32.613196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.602282 5991 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:35:32.613196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.604774 5991 fs.go:135] Filesystem UUIDs: map[0c4b3811-bf74-40ee-bd1d-6a26a3e7ae18:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ff6df988-3d6a-48a4-84df-9ece43f90eb4:/dev/nvme0n1p4] Oct 30 19:35:32.613398 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.604790 5991 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e4d50044561fb983ca790bf58221691a691e4df0cc8ef55ceece2fb4a373ee4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e4d50044561fb983ca790bf58221691a691e4df0cc8ef55ceece2fb4a373ee4/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e853cd61938490633f3ea97147c498a1005f06208457b7ac3d662b03e034a49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e853cd61938490633f3ea97147c498a1005f06208457b7ac3d662b03e034a49/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/679637e378b0fdf60731b9585d4b264710905192bd769a250320d90269ea68ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/679637e378b0fdf60731b9585d4b264710905192bd769a250320d90269ea68ab/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ae60f5e92d9a48744aeb354011e951804b0360175a1edea404752b26956c8bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ae60f5e92d9a48744aeb354011e951804b0360175a1edea404752b26956c8bd/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb/userdata/shm major:0 minor:520 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492/userdata/shm major:0 minor:513 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d/userdata/shm major:0 minor:516 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f17572b48b2653eaa77c916aee7eb8bcf5e833a2b4594c0b1ff9b6344408bc6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f17572b48b2653eaa77c916aee7eb8bcf5e833a2b4594c0b1ff9b6344408bc6f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c/userdata/shm major:0 minor:460 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~projected/kube-api-access-qb29l:{mountpoint:/var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~projected/kube-api-access-qb29l major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac523a2-773b-493f-bb76-8faa6f8c3bb1/volumes/kubernetes.io~projected/kube-api-access-wg285:{mountpoint:/var/lib/kubelet/pods/3ac523a2-773b-493f-bb76-8faa6f8c3bb1/volumes/kubernetes.io~projected/kube-api-access-wg285 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488/volumes/kubernetes.io~projected/kube-api-access-pdh6l:{mountpoint:/var/lib/kubelet/pods/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488/volumes/kubernetes.io~projected/kube-api-access-pdh6l major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~projected/kube-api-access-fvw4f:{mountpoint:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~projected/kube-api-access-fvw4f major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~projected/kube-api-access-fb57z:{mountpoint:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~projected/kube-api-access-fb57z major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/557d0d40-9bfa-4d88-9ee2-ac07635e7daa/volumes/kubernetes.io~projected/kube-api-access-6jvzt:{mountpoint:/var/lib/kubelet/pods/557d0d40-9bfa-4d88-9ee2-ac07635e7daa/volumes/kubernetes.io~projected/kube-api-access-6jvzt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f0c8ec3-e700-43b5-a948-ded642dc21fe/volumes/kubernetes.io~projected/kube-api-access-9vqfj:{mountpoint:/var/lib/kubelet/pods/6f0c8ec3-e700-43b5-a948-ded642dc21fe/volumes/kubernetes.io~projected/kube-api-access-9vqfj major:0 minor:515 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~projected/kube-api-access-h987g:{mountpoint:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~projected/kube-api-access-h987g major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/814f2828-2c09-40ea-b910-02df6029fffd/volumes/kubernetes.io~projected/kube-api-access-b49cj:{mountpoint:/var/lib/kubelet/pods/814f2828-2c09-40ea-b910-02df6029fffd/volumes/kubernetes.io~projected/kube-api-access-b49cj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~projected/kube-api-access-t4z8q:{mountpoint:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~projected/kube-api-access-t4z8q major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92a56f5b-251a-4f17-9dc8-ed6434db1328/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/92a56f5b-251a-4f17-9dc8-ed6434db1328/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:512 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~projected/kube-api-access-nbrf6:{mountpoint:/var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~projected/kube-api-access-nbrf6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~projected/kube-api-access-66wvp:{mountpoint:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~projected/kube-api-access-66wvp major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~projected/kube-api-access-5z7gx:{mountpoint:/var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~projected/kube-api-access-5z7gx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~secret/metrics-certs major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/alertmanager/9 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/kube-api-access-nvwws:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/kube-api-access-nvwws major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/tls-assets major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/config-volume major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/web-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/config-reloader/2 major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/prometheus/15 major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/kube-api-access-59zt2:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/kube-api-access-59zt2 major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/tls-assets major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/config major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/web-config major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c58f03ba-4840-4237-88ee-82bf233ac315/volumes/kubernetes.io~projected/kube-api-access-v59jq:{mountpoint:/var/lib/kubelet/pods/c58f03ba-4840-4237-88ee-82bf233ac315/volumes/kubernetes.io~projected/kube-api-access-v59jq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~projected/kube-api-access-42kqf:{mountpoint:/var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~projected/kube-api-access-42kqf major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~projected/kube-api-access-2n857:{mountpoint:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~projected/kube-api-access-2n857 major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~projected/kube-api-access-t2jtg:{mountpoint:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~projected/kube-api-access-t2jtg major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~projected/kube-api-access-72j9m:{mountpoint:/var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~projected/kube-api-access-72j9m major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~projected/kube-api-access-mm8bf:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~projected/kube-api-access-mm8bf major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad1332e-70fd-49ce-b1c2-48e50110f29b/volumes/kubernetes.io~projected/kube-api-access-rsmbd:{mountpoint:/var/lib/kubelet/pods/fad1332e-70fd-49ce-b1c2-48e50110f29b/volumes/kubernetes.io~projected/kube-api-access-rsmbd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~projected/kube-api-access-d88zq:{mountpoint:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~projected/kube-api-access-d88zq major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/78132aa64d8c3f1b358b7ae9ab0adb982b6571f10ac1dea447fd931c15cda76a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3751d84a4014ad83fc4fa7eed993f26676549fa4ba5aaff292e3cc0189087443/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c686a1df5a2af51490dd9dd90f2e072ac7adbc30437e4d976aa9b8cb81237d2a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3fe03405c0da272513f87847d6bc9f10c0b19346031356c23c8d9ad7f6a03b56/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0224810d6c44ef942992af6adafb1609cae54771e08ff7de55a095dea7731ae3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/32b417241cc319861b84e6f932a1d8b87d34e465a8a321306a8fbc090650d59a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/09a27dc5e22fe6e0af97f6965ff0663a19f437c43db89bb7e360ab0cbaf6f54b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c2331426b523ac7376184272045ef37f38f1a26161ab5b1ac7ceb46951b90ae0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2db084d2c123995e588321ba4a14e4f2447f7f9f70f5a0070a12e6db9c23edb5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/4bd4df7100b2fed7a0fee9874bb2c784afaa8fc5aa5c431951061f2c791cb0bd/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/7ec7eebf14f027cef8e5fd9e6bd4c750cda2761a13ee37e159723bac8cfabdff/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/6c64750f551d7dc957a7f1512d120659c51d99bc58ef3e0b739d436b134cace9/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a8e9b72a0bc1396704f3776ca1d4c18edf0466f6c7db855938b1af14b30d391c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cfa1a98470a556ee9463549b491c00fdbe47e824e32f4bcefb3b3d9a6129068f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b70fef7790c99e40988b84fad338a4046a9e70fa98737aa399d35cf7ab5044a7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e4bbc7a1e41150406bb8077535a137ddfa4efbebe00e83441516b39f3b3da9b1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/97d4c554ca2275982033ae5c951c46187524af65226358b7dc9b9db3184819f0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fba5bf33d95716f134d772fca9fabcf9c2348c320967e69a774b0a40a2b76324/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5035e6a1f4289ad1b8a3f06c3ac71eea82bee61e89b56f06daaea1a68cdc9ad6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f76bdfa60e4f707b4ee9234821b5264c4d7113ff836270d89fbdc75f23625b5c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/25d54ffa060cdff6f064e710c9fa111e54978da986296728e57b84520ffd4569/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b19e6cf8ac14f6eb0778913516cb2c96b2af1835dcb3601517f3900ec461d6e9/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e9012af0ab027de8e247cad0bba81111c7762d436429a9ac8b90a7bb2ddf57bc/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0484b7daa29f1ffc6f651203a7a75fa07a330ed4e1756a46b6833cc4b65f1457/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/69b7002f86a7bd8bc5126f11179e668dfaa645ff6f725f0f3248e2ddad3eaa73/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/423e53b1f895cad6280651d5a6db5ba98ff734dd65f996c412a122150dfc88e4/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/855dc15c6b64af0e37d3af06aff90f1132be5df086501147ac78f1d1f513c93b/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/43f5fa345a23ea9c9681d08c6628b8abfc4998c89b39cc66fa4e223b8986b5cb/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/d866ffc84b50e311726b188ad2b84926dbb837ca30b3d1eea8de6a028dec0019/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/2d33e9767b2ee4047ae3bb15f8013b79781eb27e81a03b5783e35b7eb38efa3b/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/5a3c6bb353174e85a5ea25bac635bc6a199bf7a376823e67d3fc7ac94db48234/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5a3e1bd58852b0c0690cb1e87a9828a2afce103a2a888f0b2b801a51b82320a7/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/d22bb52922136038c5f1b987e0f768164ca1428421fbcda7818006e4fe140ff8/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/198c38bd3171c2d345e845d0ea72411425bb76cc1e0e1e03c43c3d809867dd13/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/ee64fe31c2f8a45fd7ee050066fc4d789740094696fac32efe57a997782633bf/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/39849fe631303c9eef65399a2205934c849f11f5ed12e48bd81344254821ebe9/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/5cfe7478ea5d6762bb663936f8cf9e0d934ac2ee794daba0531b784349ef226b/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/3e01b325bdb63f7d9a1396dcfdca8c1aa4d008a76b75476f96ad60bf4f0a69f0/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/18168adeaff213290bbe80c3704d6a0b05ecdf9f60b90a81448a9e53bb81237f/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/da964d0d90b22dc497d4a924cd4d17d7422d301545a21a11037a020c6a9de473/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/c8c47cce6429ad2d42e24d20379139a0e871121f091370b6fb41511f730862df/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/0503e275f89ecaaa7eff7ce73892b25ec18eeb86d290a894226272854e3725dd/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/38036a3e2b7464a9e2e3fbc122eca4add11e77e4677bf8babac99c112e11c845/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/783fa07e3c5762c8c941d64c69b2d265e8790661295069d36d7c2be45b7ff278/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/02aa91f3aabb78c6a508eb0a039237cd411f9ce4349657e8c0a7da047afbc40f/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/9b6254b9479071cebfb9e95249e5ea61db812d6a2bf51624b99a4aee8289ef56/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4c9b751818d50d4381a39d2bd234c2bae18fc0fb0083daab0101b570929fc344/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/055aeae303228e7cd3a120690344feaec3c1a4174783c75c7dceb207967dbf5d/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/2bc1126f32d6c80dd03768f7d556b7eda01728c5dc1fc0417202dc1cf6d4c00d/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/dda3051d6b028371714e69348293f2ee623646d2bf9d9cde44e74592e852fe0e/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/e9a7a0d66bb3b32d0fddbd9326d63c7e38c62cd10dffdd5c2f551701e87c5c31/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/5f363766b80da7107c8edb10b1869684e84831f95b4d155d15e036f6b2a51a66/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/f130b2e35ed4df04e7a0b627d5ced1875862df922d5c5d75b5f515f93d3fae60/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/d7974bbf9cd078b049d83aa7b53fc97bf311a1ef4dc43f7e40255cf2dfe28f7c/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/209ee76046844b4d3d39a698a1bd32d45670333a71019c06646f3710c95979b8/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/09872f28851b696bc407d0c20e518e0887bf0eaf3ff9acea5a1c71c101d5088c/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/30aeec45e389e046452954c8c59923e4da33be159356317b99b1f5e8dbd38e8f/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/e6c1f8492a10fa9ba8d640630f3ce400d4650a26ee61867b19c419c8cff25bdc/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/e356652b91f48b6b805c79a39c536b0faac36695a9e8087123253a8f9a859176/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/d9048b67cbe9a44845eccb2d01ae9a86bf7766e3d702eed15ea50928a91b3261/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/1632199822da30190bb484d329331a55c2f1c85d71f5a543cc1b812fa3f75877/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/2db9f7c6f664beb3e6cd816da909fae8c7c13bd34508ee7abe8d271780f79254/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/e1f3a1c7aec6c91bb75c813a8e1c23d40850baa7805f19b9559b2f8bca2c57bf/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/08ae2a9849444adceffde1fd8b34c827b85359f4c0fb8e79aca0574b91627806/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/f8a4f1f2ebfe9d8f3d243b1671ea357f7dab7f8b66faa6b3c220187a8b436531/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/6625e39cf1de5f7ae52a22a0ff8a99ebab853a7dd38c05057a0e35572dc7c567/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/1abaf5bcfc1b231211fc735d32e1402f50c68d0a4a07518f335653df0daf5357/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/13fbcfb9966ed91ac5f0eb0a5db9a2724576d217561710750fbd2e8a2a2f3523/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/1c68bf4ccc386a34b27615c6b8f205f9b4d584c2464b2b1eeb9ecacb62304ee7/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/0ba07434d59d2ac20a59f48d0b7fb3b08bccc8e2eef71d44187ef057b1bfb15d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/e5fd0fc1cdcb5d81f7e544b957ffe7b3fbcb0dbc20e355c556fe25278d539cce/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/c7828004114ca011b78c914e3794805300d0b4e774ec5d92e2d3519ea97b3203/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/ebc50c03143756a15f5d9c8e4b62611ed7ab8f699ea0cdf539fb71c3e87c4f92/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/c69beb5e451ab42bbbb982823a928fa5d1447fd4c2ffb9ddebe5a0376163a324/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/b734f33704430c02656c89bed584e0d1477757a60aa4bcf169c648cde2361f00/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/b2c1f35f01bc91c923c19c0bce980b6ef37ce535567a3c7fbc9a97ec1c4b303f/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/750dc3ac32dcabb9b5c69d5c851001da3294f3aca25e73553575930682ed1010/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/35c6073f4f92ae1fe0bb1805732242b05f9f53498b4cbf906182d92918b7bc26/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f165f88298b9c0bd6c161af04eecb513d79dfcad1d0d4df8793dcf393a1e1c37/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/08f02881b616a5df9026de86edf361b03786beb8e79fc4962c46ad49be4ef7de/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/de66b3ced2a2374769dee0916f2db3ee805cf6b4f4000fd0c7d84db25fc7fd99/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/20abf6f2edf5475f8b6358794bef6e0794d95d78ed6a7ffee53f00905af480e9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e5d399b4f6beba9e22a9692e81fbb6d1b49e3e6b0d5a84834b4e245aa0bc18ff/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/de038490061bdcf3c9a4f90b260609a0be801e6ccda4d1d50d8a686d54bebc39/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/9b6653f08b19f0f50aa1ddcea96aa9f5a5f806f092942e3a86b7dfa579b3ded0/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/030e08b51189a5b7eea71080899bc1485b2ad4fb30d5b868c6f2051fdf4043d8/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/73362970d53860fcfe54c21629ce11b5c5ec3f42591f6653314e63ec6363b9a5/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8fb707d82c555928cd0bdb0d4302138d9448febb16f3ed2c36f1751fec8cf424/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/881af22f997c50ee09640fb12a7daf88aa7023f1189dcafe7a63b2d3f54bdddb/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:35:32.616878 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.616409 5991 manager.go:217] Machine: {Timestamp:2026-10-30 19:35:32.615891071 +0000 UTC m=+0.065383787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099643 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be418399c6e50cde7cc0dcacd0761 SystemUUID:ec2be418-399c-6e50-cde7-cc0dcacd0761 BootID:40c49548-cb19-4cc8-85d2-2793bbcebe88 Filesystems:[{Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0841e45a8e4ae9bfcccd011ac2e66031bafbbc5ba78a2c0453a682238c2b421/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc8739c3e3878e191ee55daa63b9b2b4acd2b508ae4d48f87c981f19f7344893/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92a56f5b-251a-4f17-9dc8-ed6434db1328/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:512 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6f0c8ec3-e700-43b5-a948-ded642dc21fe/volumes/kubernetes.io~projected/kube-api-access-9vqfj DeviceMajor:0 DeviceMinor:515 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc60649b29178fb125e440e9950d079239c32e410a75b58a432e6b812a97d27f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36f414b936400405bbb00c746edb8c5387bb6fd2630a8b5fc0f251a7a82d75d4/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488/volumes/kubernetes.io~projected/kube-api-access-pdh6l DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39946477fee6b2a0f7a2fd55dcce56fa766247895a912a6493bfa331bfd36f6c/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:408 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~projected/kube-api-access-t4z8q DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:409 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f17572b48b2653eaa77c916aee7eb8bcf5e833a2b4594c0b1ff9b6344408bc6f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/kube-api-access-nvwws DeviceMajor:0 DeviceMinor:305 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:342 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c/userdata/shm DeviceMajor:0 DeviceMinor:460 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6d1d26ad22afc4c7f4545adf543d5b9bfc6d1ca965d0890a9d3411e5a8107dd/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/908aaccc-1937-4ac9-8614-28365a9fb984/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~projected/kube-api-access-nbrf6 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fef780a60f5fee68d23285d3de1e162f737f1433909c98a76b563acb1822215/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:458 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/kube-api-access-59zt2 DeviceMajor:0 DeviceMinor:459 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~projected/kube-api-access-fvw4f DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5dfa0247d75f5d41c3e8cee1b987dc7d58066883b35763286df7f180aa1c632/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:370 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:402 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~projected/kube-api-access-72j9m DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc76c3f5-d680-4468-a8ae-b100e41e253c/volumes/kubernetes.io~projected/kube-api-access-t2jtg DeviceMajor:0 DeviceMinor:411 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4009f08d-95f6-4e73-ba56-2740e4e55336/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/557d0d40-9bfa-4d88-9ee2-ac07635e7daa/volumes/kubernetes.io~projected/kube-api-access-6jvzt DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e300e576-627d-46ea-a62d-567ec3f8d5c3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/volumes/kubernetes.io~projected/kube-api-access-42kqf DeviceMajor:0 DeviceMinor:372 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:455 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/997ee0b265213d9620466a9905aae3ee0813471635aedddd107e0827c13469eb/userdata/shm DeviceMajor:0 DeviceMinor:520 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/106417f444cbfcd165fd73ac37a43f829049c44b7ba46c3b07ff3ecd85e5fb71/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~projected/kube-api-access-fb57z DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/868f14de3f5719bad25e27aa1b5b8cc013439d8fa98bec5ccdda4ccd09627e38/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac523a2-773b-493f-bb76-8faa6f8c3bb1/volumes/kubernetes.io~projected/kube-api-access-wg285 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98c38cf2-9e21-4d7f-836b-9983359012fc/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ecb26d-be82-4455-8dae-e0b7489350ee/volumes/kubernetes.io~projected/kube-api-access-2n857 DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7977864-d9ee-43d9-9c44-6611a31dba60/volumes/kubernetes.io~projected/kube-api-access-mm8bf DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:444 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:456 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fad1332e-70fd-49ce-b1c2-48e50110f29b/volumes/kubernetes.io~projected/kube-api-access-rsmbd DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4613545a7466b17f207fdb8e4935fd2250da52d1c3db2a5c317154d8b0ee8f4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd903c505629257cd29d30e800579cbe24d62f7ef14c6e0dab538d69294b6699/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~projected/kube-api-access-66wvp DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:298 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae28a49963298013be6708b7c8cbf72d27cf1a0fe17183d780cb3ffb33d4f492/userdata/shm DeviceMajor:0 DeviceMinor:513 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e4d50044561fb983ca790bf58221691a691e4df0cc8ef55ceece2fb4a373ee4/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c58f03ba-4840-4237-88ee-82bf233ac315/volumes/kubernetes.io~projected/kube-api-access-v59jq DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~projected/kube-api-access-5z7gx DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:452 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:457 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/814f2828-2c09-40ea-b910-02df6029fffd/volumes/kubernetes.io~projected/kube-api-access-b49cj DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/679637e378b0fdf60731b9585d4b264710905192bd769a250320d90269ea68ab/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/140004b8-1074-4ef5-a31e-9c9f52c46d8a/volumes/kubernetes.io~projected/kube-api-access-qb29l DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:451 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0797257-801e-4dac-b7ea-ec05f4a382f3/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:508 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e853cd61938490633f3ea97147c498a1005f06208457b7ac3d662b03e034a49/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/volumes/kubernetes.io~projected/kube-api-access-d88zq DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbe808e95d6e7e7ac3a06c24a4321f5229fb72b6b91aca175422c2c12e81ae6d/userdata/shm DeviceMajor:0 DeviceMinor:516 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/724b9880-4314-41ad-93f7-40eb026a0fb2/volumes/kubernetes.io~projected/kube-api-access-h987g DeviceMajor:0 DeviceMinor:254 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7173b944a16915e8767bd366572a7053a47eb7e6d37a363520408e718e041d9/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:450 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99ab51f7-f13e-41df-985a-172ac7fe4927/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ae60f5e92d9a48744aeb354011e951804b0360175a1edea404752b26956c8bd/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e4d50044561fb9 MacAddress:9a:f3:5f:28:9d:5f Speed:10000 Mtu:8901} {Name:0e853cd61938490 MacAddress:e6:48:af:da:83:1e Speed:10000 Mtu:8901} {Name:36f414b93640040 MacAddress:5a:34:fd:17:d3:91 Speed:10000 Mtu:8901} {Name:39946477fee6b2a MacAddress:7a:72:6d:32:30:7f Speed:10000 Mtu:8901} {Name:6ae60f5e92d9a48 MacAddress:72:47:a8:4a:49:90 Speed:10000 Mtu:8901} {Name:74ce61a1d1aa93f MacAddress:92:e1:28:33:40:1f Speed:10000 Mtu:8901} {Name:7fef780a60f5fee MacAddress:9e:8e:d3:5d:9c:15 Speed:10000 Mtu:8901} {Name:868f14de3f5719b MacAddress:da:5a:e7:2c:23:d7 Speed:10000 Mtu:8901} {Name:997ee0b265213d9 MacAddress:be:47:de:4f:07:5f Speed:10000 Mtu:8901} {Name:ae28a4996329801 MacAddress:1e:52:e4:d8:4b:32 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:d4:22:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:76:25:07:84:62:71 Speed:0 Mtu:8901} {Name:c6b773be3270ead MacAddress:76:9c:aa:de:3f:97 Speed:10000 Mtu:8901} {Name:dbe808e95d6e7e7 MacAddress:2e:41:e1:89:9d:16 Speed:10000 Mtu:8901} {Name:e5dfa0247d75f5d MacAddress:1e:e9:31:1b:2d:5b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:d4:22:49 Speed:0 Mtu:9001} {Name:f7173b944a16915 MacAddress:22:9c:9b:c4:b3:4e Speed:10000 Mtu:8901} {Name:fb973f7e4a5e2f1 MacAddress:42:55:ad:e6:0f:ba Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7e:1e:30:71:76:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:93:e7:61:dc:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:35:32.617192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.616882 5991 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:35:32.617192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.616936 5991 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:35:32.617352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617322 5991 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:35:32.617541 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617354 5991 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:35:32.617596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617544 5991 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:35:32.617596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617551 5991 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:35:32.617596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617562 5991 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:35:32.617596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617583 5991 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:35:32.617711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617688 5991 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:35:32.617920 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.617910 5991 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:35:32.618043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618034 5991 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:35:32.618084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618075 5991 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:35:32.618150 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618090 5991 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:35:32.618150 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618108 5991 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:35:32.618150 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618118 5991 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:35:32.618150 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.618128 5991 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:35:32.619549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.619531 5991 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:35:32.619549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.619549 5991 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:35:32.620217 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.620189 5991 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:35:32.624354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624338 5991 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:35:32.624354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624354 5991 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:35:32.624485 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624471 5991 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:35:32.624673 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624661 5991 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:35:32.624707 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624691 5991 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:35:32.624795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624785 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624800 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624806 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624813 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624818 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624824 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624832 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:35:32.624840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624840 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624847 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624854 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624862 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624871 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624900 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:35:32.625087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.624909 5991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:35:32.629298 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.629276 5991 server.go:1263] "Started kubelet" Oct 30 19:35:32.629434 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.629331 5991 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:35:32.630051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.629557 5991 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:35:32.630051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.629654 5991 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:35:32.630051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.629989 5991 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:35:32.629869 ip-10-0-132-4 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:35:32.630699 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.630680 5991 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633310 5991 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633337 5991 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633500 5991 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633511 5991 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633549 5991 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:29:23 +0000 UTC" deadline="2028-04-30 22:44:46.128091561 +0000 UTC" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633573 5991 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13155h9m13.494522541s" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.633620 5991 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:35:32.634122 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.633627 5991 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found" Oct 30 19:35:32.637000 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.636140 5991 factory.go:55] Registering systemd factory Oct 30 19:35:32.637000 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.636165 5991 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:35:32.637294 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637259 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557d0d40-9bfa-4d88-9ee2-ac07635e7daa" volumeName="kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637297 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637312 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637325 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637337 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637349 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:35:32.637409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637405 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637418 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0797257-801e-4dac-b7ea-ec05f4a382f3" volumeName="kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637434 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637447 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637473 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637495 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0797257-801e-4dac-b7ea-ec05f4a382f3" volumeName="kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637509 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637521 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637567 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637582 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" volumeName="kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637595 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637606 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637618 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637630 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba" volumeName="kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637642 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637648 5991 factory.go:153] Registering CRI-O factory Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637653 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637665 5991 factory.go:223] Registration of the crio container factory successfully Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637668 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:35:32.637687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637680 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac523a2-773b-493f-bb76-8faa6f8c3bb1" volumeName="kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637702 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637716 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908aaccc-1937-4ac9-8614-28365a9fb984" volumeName="kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637723 5991 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637730 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637742 5991 factory.go:103] Registering Raw factory Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637743 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e" volumeName="kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637758 5991 manager.go:1199] Started watching for new ooms in manager Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637763 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e300e576-627d-46ea-a62d-567ec3f8d5c3" volumeName="kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637776 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637790 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637803 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" volumeName="kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637821 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow" seLinuxMountContext="" Oct 30 19:35:32.638551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.637836 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.638975 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639010 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" volumeName="kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639024 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557d0d40-9bfa-4d88-9ee2-ac07635e7daa" volumeName="kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639075 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639090 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639112 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639130 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639142 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:35:32.639173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639166 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba" volumeName="kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639180 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="140004b8-1074-4ef5-a31e-9c9f52c46d8a" volumeName="kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639202 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba" volumeName="kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639218 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4009f08d-95f6-4e73-ba56-2740e4e55336" volumeName="kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639238 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639258 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58f03ba-4840-4237-88ee-82bf233ac315" volumeName="kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639275 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639307 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba" volumeName="kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639322 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908aaccc-1937-4ac9-8614-28365a9fb984" volumeName="kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639337 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639357 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639395 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639415 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7" volumeName="kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639431 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639470 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639493 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639509 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639528 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:35:32.639566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639543 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4009f08d-95f6-4e73-ba56-2740e4e55336" volumeName="kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp" seLinuxMountContext="" Oct 30 19:35:32.640260 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.639918 5991 manager.go:319] Starting recovery of all containers Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642509 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642542 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642556 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e300e576-627d-46ea-a62d-567ec3f8d5c3" volumeName="kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642593 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad1332e-70fd-49ce-b1c2-48e50110f29b" volumeName="kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642613 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642633 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642646 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642663 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908aaccc-1937-4ac9-8614-28365a9fb984" volumeName="kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642675 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908aaccc-1937-4ac9-8614-28365a9fb984" volumeName="kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642702 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c38cf2-9e21-4d7f-836b-9983359012fc" volumeName="kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642719 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e300e576-627d-46ea-a62d-567ec3f8d5c3" volumeName="kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642732 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="140004b8-1074-4ef5-a31e-9c9f52c46d8a" volumeName="kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642749 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642761 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642774 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642793 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac523a2-773b-493f-bb76-8faa6f8c3bb1" volumeName="kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642804 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4009f08d-95f6-4e73-ba56-2740e4e55336" volumeName="kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642821 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642834 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642845 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642863 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.642876 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0c8ec3-e700-43b5-a948-ded642dc21fe" volumeName="kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643164 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a56f5b-251a-4f17-9dc8-ed6434db1328" volumeName="kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643250 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643277 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643324 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643336 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643496 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e" volumeName="kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643512 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e300e576-627d-46ea-a62d-567ec3f8d5c3" volumeName="kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643529 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643541 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724b9880-4314-41ad-93f7-40eb026a0fb2" volumeName="kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643556 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643568 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643577 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857" seLinuxMountContext="" Oct 30 19:35:32.643569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643592 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908aaccc-1937-4ac9-8614-28365a9fb984" volumeName="kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643616 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643657 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643683 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad1332e-70fd-49ce-b1c2-48e50110f29b" volumeName="kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643706 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="814f2828-2c09-40ea-b910-02df6029fffd" volumeName="kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643728 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ab51f7-f13e-41df-985a-172ac7fe4927" volumeName="kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643737 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643755 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643763 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c38cf2-9e21-4d7f-836b-9983359012fc" volumeName="kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643771 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7" volumeName="kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643785 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643804 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c38cf2-9e21-4d7f-836b-9983359012fc" volumeName="kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643907 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58f03ba-4840-4237-88ee-82bf233ac315" volumeName="kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643949 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc76c3f5-d680-4468-a8ae-b100e41e253c" volumeName="kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643975 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.643987 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644018 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5533697-e0c7-42b4-958f-cbfc3541bc3e" volumeName="kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644095 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ecb26d-be82-4455-8dae-e0b7489350ee" volumeName="kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644106 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c38cf2-9e21-4d7f-836b-9983359012fc" volumeName="kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644131 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644172 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7977864-d9ee-43d9-9c44-6611a31dba60" volumeName="kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644182 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad1332e-70fd-49ce-b1c2-48e50110f29b" volumeName="kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644213 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4450c8f2-6d00-4a2d-aee2-4acd31855fea" volumeName="kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644223 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4395c20-e68a-4714-99e3-6992284251f6" volumeName="kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644237 5991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" volumeName="kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644247 5991 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:35:32.644692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.644254 5991 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:35:32.653492 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.653458 5991 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:35:32.669255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.669223 5991 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:35:32.669680 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.669662 5991 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:35:32.669828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.669820 5991 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:35:32.669976 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.669932 5991 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:35:32.672250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.672212 5991 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:35:32.672428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.672388 5991 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:35:32.678980 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.678948 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:35:32.679400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.679162 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:35:32.679702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.679686 5991 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:35:32.683652 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.683623 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="b13258187328279b1f5d16c20293e4f174f3583fe53b66cbc8c729031fc78ee3" exitCode=0 Oct 30 19:35:32.691429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.691408 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:35:32.691628 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.691612 5991 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:35:32.691813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.691788 5991 generic.go:356] "Generic (PLEG): container finished" podID="908aaccc-1937-4ac9-8614-28365a9fb984" containerID="ca8145735c42cab33b0aac9fa9c25a1886bf7c009215a35b0266afb42dfbacfe" exitCode=1 Oct 30 19:35:32.692791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.692767 5991 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:35:32.711462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.711419 5991 generic.go:356] "Generic (PLEG): container finished" podID="d5ecb26d-be82-4455-8dae-e0b7489350ee" containerID="55ea5e63c1614417d53c543fe347dd74bd2da62b6501e4a531af871b90e807d7" exitCode=0 Oct 30 19:35:32.720056 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.720022 5991 generic.go:356] "Generic (PLEG): container finished" podID="5dff8c1e3d20edc13c5947130e8a5c14" containerID="112ff42866bd54eba2419081a3b93f49b5bacfdce5a496f0052298978116945d" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736453 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="82bce8838ab7fcbd61a8f4c171aa37b82d4aaf55bb44defcfd29e5d683673a0d" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736482 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="294e5ad1b0bce64e1f24e4d9cdacaf1b4ba6265140cc31254438e8270ebbe41c" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736493 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="fd3bee334fb0a5ec3139f2b80b36acd3bc456f90c8053c1591a7cc5ef9b1de9d" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736501 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="27e75cb877c2a45f92ca922ce87276cfa996ec143a66037113e66bff1f01f430" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736509 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="ffadf5a42a961b36ed3eb1670df703d731e038f948e798dd88cab318dd15f12d" exitCode=0 Oct 30 19:35:32.736535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.736516 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488" containerID="7de76715ddcb647fdf1014965e1c5778adfa25fabdb775105f946f3a6deff249" exitCode=0 Oct 30 19:35:32.744214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.744184 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" exitCode=0 Oct 30 19:35:32.770112 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.770083 5991 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:35:32.773003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.772982 5991 manager.go:324] Recovery completed Oct 30 19:35:32.803413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803379 5991 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:35:32.803413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803407 5991 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:35:32.803608 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803453 5991 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:35:32.803608 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803591 5991 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:35:32.803668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803601 5991 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:35:32.803668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803652 5991 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:35:32.803668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803658 5991 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:35:32.803668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803664 5991 policy_none.go:50] "Start" Oct 30 19:35:32.803778 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803681 5991 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:35:32.803778 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803694 5991 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:35:32.803865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803853 5991 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:35:32.803894 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803866 5991 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:35:32.803894 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.803872 5991 policy_none.go:44] "Start" Oct 30 19:35:32.809090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.809052 5991 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:35:32.813872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.813852 5991 manager.go:341] "Starting Device Plugin manager" Oct 30 19:35:32.814347 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.813878 5991 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:35:32.814347 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814345 5991 server.go:87] "Starting device plugin registration server" Oct 30 19:35:32.814643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814630 5991 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:35:32.814704 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814644 5991 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:35:32.814785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814759 5991 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:35:32.814954 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814917 5991 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:35:32.815084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814960 5991 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:35:32.815084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.814966 5991 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:35:32.815084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.815017 5991 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:35:32.814948087Z","UUID":"a93b8afd-8053-456a-bc90-a5b9f34229ca","Handler":null,"Name":"","Endpoint":""} Oct 30 19:35:32.816793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.816766 5991 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:35:32.817081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.816814 5991 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:35:32.817150 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.817082 5991 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:35:32.915188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.915151 5991 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:35:32.918676 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.918652 5991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:35:32.918801 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.918690 5991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:35:32.918801 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.918705 5991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:35:32.918801 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.918781 5991 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.948449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.948424 5991 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.948600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.948503 5991 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.970331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.970283 5991 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"] Oct 30 19:35:32.971025 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971010 5991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.971130 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971089 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"e6ff4aa892aaddca7da54639fb3560aefa1bd93229f69055c0705b8b87b6d43f"} Oct 30 19:35:32.971164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971151 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerDied","Data":"112ff42866bd54eba2419081a3b93f49b5bacfdce5a496f0052298978116945d"} Oct 30 19:35:32.971194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971170 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"fb1a8ebd169240edce0a889f2caf80b57f50316adfb041687be3cde64a0225fb"} Oct 30 19:35:32.971242 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971194 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"ea90a98b8fdfc8add1b448cc029d8020d02a4177d87936f4126db2406508cdb1"} Oct 30 19:35:32.971242 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971211 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"db816830b24e6ebe4ef0bec1677df5ee3758545224ac671181b1066a7f65687e"} Oct 30 19:35:32.971345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.971333 5991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.993703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.993621 5991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:35:32.993703 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.993676 5991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:32.994000 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:32.993981 5991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:35:32.994062 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:32.994038 5991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.073352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.073312 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.073352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.073352 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.073567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.073403 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.173975 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.173939 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.173975 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.173975 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.174201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.174007 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.174201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.174044 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.174201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.174074 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.174201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.174127 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.618934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.618904 5991 apiserver.go:52] "Watching apiserver" Oct 30 19:35:33.676839 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.676311 5991 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:35:33.677938 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.677901 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84748fcdfb-kqjdr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws","openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks","openshift-network-operator/iptables-alerter-twm5d","openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z","openshift-dns/dns-default-pgmdj","openshift-multus/multus-t5d7p","kube-system/konnectivity-agent-l7lfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-console/console-7574f7c6c-ksbt6","openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j","openshift-image-registry/node-ca-ftqg8","openshift-monitoring/metrics-server-695b5bb69b-wlqqm","openshift-insights/insights-runtime-extractor-ts96r","openshift-monitoring/node-exporter-csl8b","openshift-multus/network-metrics-daemon-8lk6r","openshift-dns/node-resolver-hmsqs","openshift-network-diagnostics/network-check-target-xspfd","kube-system/global-pull-secret-syncer-zzgh7","openshift-ingress-canary/ingress-canary-p24hl","openshift-multus/multus-additional-cni-plugins-wksc9","openshift-ovn-kubernetes/ovnkube-node-k5fx8","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-747576895b-zqwt6","openshift-cluster-node-tuning-operator/tuned-s9qdt","openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:33.692200 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692162 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.692461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692431 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.692461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692444 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\"" Oct 30 19:35:33.692619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692472 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.692619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692532 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.692619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692561 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:35:33.692619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.692617 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\"" Oct 30 19:35:33.693175 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.693147 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:35:33.693261 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.693147 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.693261 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.693153 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\"" Oct 30 19:35:33.696671 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.696648 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.696958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.696938 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.697717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.697695 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\"" Oct 30 19:35:33.697873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.697840 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:35:33.698147 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698120 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:35:33.698769 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698746 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\"" Oct 30 19:35:33.698769 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698761 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\"" Oct 30 19:35:33.698918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698791 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:35:33.698918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698872 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:35:33.698918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698880 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:35:33.698918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.698888 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.699092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699004 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\"" Oct 30 19:35:33.699092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699056 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:35:33.699092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699056 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.699092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699083 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.699262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699147 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:35:33.699262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699197 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:35:33.699262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699246 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\"" Oct 30 19:35:33.699431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699298 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.699431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699346 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.699537 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699522 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:35:33.699588 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699537 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.699639 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699603 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:35:33.699721 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699704 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.699721 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699704 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:35:33.699836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.699712 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.703301 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.703267 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\"" Oct 30 19:35:33.703383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.703373 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:35:33.703434 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.703404 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\"" Oct 30 19:35:33.704090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.704079 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.736852 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.736824 5991 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:35:33.748527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.748493 5991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.748717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.748633 5991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.771568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.771537 5991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:35:33.771711 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.771594 5991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.772617 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.772603 5991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:35:33.772661 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.772636 5991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" Oct 30 19:35:33.777659 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777619 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmbd\" (UniqueName: \"kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.777659 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777650 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59jq\" (UniqueName: \"kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777668 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777686 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8bf\" (UniqueName: \"kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777707 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777723 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88zq\" (UniqueName: \"kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777747 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777773 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777804 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777829 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777854 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb29l\" (UniqueName: \"kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:33.777882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777879 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777904 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777932 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.777957 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778032 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778083 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778120 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778140 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778156 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908aaccc-1937-4ac9-8614-28365a9fb984-ovn-node-metrics-cert\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778185 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778205 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778242 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778263 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778303 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778385 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvzt\" (UniqueName: \"kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778411 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.778429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778431 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7gx\" (UniqueName: \"kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778454 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778500 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778562 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778594 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778626 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778678 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778723 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778728 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778765 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.778806 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778843 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778875 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778879 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.778948 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.278909738 +0000 UTC m=+1.728402448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778948 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e300e576-627d-46ea-a62d-567ec3f8d5c3-data-volume\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.778971 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779005 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779054 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779089 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.779113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779115 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779179 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779264 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779301 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779322 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779346 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779410 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779456 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779521 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779532 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-script-lib\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779553 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779574 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779589 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7977864-d9ee-43d9-9c44-6611a31dba60-tmp\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779589 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779637 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779671 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779725 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779737 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58f03ba-4840-4237-88ee-82bf233ac315-tmp-dir\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.779761 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779767 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779806 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:33.779812 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.779815 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.279801584 +0000 UTC m=+1.729294288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779837 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779856 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779876 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779914 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779947 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kqf\" (UniqueName: \"kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779976 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72j9m\" (UniqueName: \"kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.779976 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-serviceca\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780016 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780050 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780088 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780133 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h987g\" (UniqueName: \"kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780168 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780198 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n857\" (UniqueName: \"kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780228 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780253 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780264 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780281 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780288 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-konnectivity-ca\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780307 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780336 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jtg\" (UniqueName: \"kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780378 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780405 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780433 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780459 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780497 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-ovnkube-config\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780502 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780543 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.780571 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780583 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.780601 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780603 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.780645 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.280632012 +0000 UTC m=+1.730124753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.780677 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.280668232 +0000 UTC m=+1.730160935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780696 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780723 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780739 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780757 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.780836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780835 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780879 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780884 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780910 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780936 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.780996 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781031 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781068 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781090 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781107 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781124 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781130 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781143 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781168 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781212 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781238 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781259 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781259 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-tuned\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781280 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781330 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781400 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781410 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-binary-copy\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.781422 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781453 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.781472 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.281457221 +0000 UTC m=+1.730949945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781508 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781559 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781571 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908aaccc-1937-4ac9-8614-28365a9fb984-env-overrides\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781588 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781606 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781661 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781697 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781726 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781756 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781783 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781822 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781850 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781879 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781911 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781931 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0797257-801e-4dac-b7ea-ec05f4a382f3-metrics-certs\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781941 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781968 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.781995 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782021 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782053 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782086 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49cj\" (UniqueName: \"kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782115 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782124 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009f08d-95f6-4e73-ba56-2740e4e55336-tmp\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782142 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782168 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782168 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/724b9880-4314-41ad-93f7-40eb026a0fb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782199 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782227 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782256 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh6l\" (UniqueName: \"kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782284 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782309 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782313 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-cni-binary-copy\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782336 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782354 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782397 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782415 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782436 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782464 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782483 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782500 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782519 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.782486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782538 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782557 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z8q\" (UniqueName: \"kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782557 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140004b8-1074-4ef5-a31e-9c9f52c46d8a-cert\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782574 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg285\" (UniqueName: \"kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782596 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw4f\" (UniqueName: \"kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782623 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782640 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782659 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782685 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782700 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782719 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782735 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782750 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782765 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782783 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782800 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782817 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782832 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782837 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e-agent-certs\") pod \"konnectivity-agent-l7lfb\" (UID: \"bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e\") " pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782851 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782869 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrf6\" (UniqueName: \"kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782887 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782904 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.782921 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783010 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783353 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wvp\" (UniqueName: \"kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783388 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783408 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783429 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783446 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783464 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783482 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783498 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783533 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783572 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783602 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783659 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-iptables-alerter-script\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783848 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99ab51f7-f13e-41df-985a-172ac7fe4927-audit-log\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.783875 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a56f5b-251a-4f17-9dc8-ed6434db1328-original-pull-secret\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.784026 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-textfile\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.784224 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-daemon-config\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.785068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.784490 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c38cf2-9e21-4d7f-836b-9983359012fc-tmp-dir\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.792819 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.792781 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 19:35:33.792956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.792891 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:35:33.793120 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793100 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:35:33.793193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793158 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:35:33.793291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793231 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:35:33.793291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793225 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.793554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793534 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 19:35:33.793554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.793548 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:35:33.794316 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794299 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.794510 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794491 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.794571 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794526 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zprrl\"" Oct 30 19:35:33.794655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794637 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:35:33.794655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794650 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:35:33.794750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794664 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:35:33.794868 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794855 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:35:33.794910 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794898 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:35:33.795009 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.794998 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 19:35:33.795101 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795087 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:35:33.795487 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795443 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:35:33.795487 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795472 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:35:33.795578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795476 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\"" Oct 30 19:35:33.795578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795477 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:35:33.795578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795555 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:35:33.795756 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795728 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:35:33.795833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795754 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:35:33.795833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795774 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:35:33.795833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795775 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:35:33.795833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795829 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:35:33.795972 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.795856 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796177 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796210 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796300 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796319 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796340 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796348 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vmphl\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796358 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796396 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:35:33.796471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796430 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-87vg7bm5h2rog\"" Oct 30 19:35:33.796849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796542 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 19:35:33.796849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796602 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:35:33.796956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796912 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgcwn\"" Oct 30 19:35:33.796956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796916 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:35:33.797041 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.796997 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j6mrb\"" Oct 30 19:35:33.797083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797048 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:35:33.797083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797072 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p99nr\"" Oct 30 19:35:33.797083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797074 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797094 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fc58z\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797124 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797126 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797146 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797154 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:35:33.797185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797173 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:35:33.797482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797185 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:35:33.797482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797202 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9e0lqdo2eio9\"" Oct 30 19:35:33.797482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797238 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.797482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797249 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:35:33.797482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797252 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:35:33.797873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797853 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:35:33.797995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797976 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dpldx\"" Oct 30 19:35:33.798078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798056 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.798128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798103 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798350 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798358 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.797855 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798541 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798742 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.798797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.798794 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-serving-certs-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.799207 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799081 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.799207 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799158 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\"" Oct 30 19:35:33.799207 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799186 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:35:33.799499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799202 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.799499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799353 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:35:33.799499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799358 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.799499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799455 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\"" Oct 30 19:35:33.799499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799496 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.799728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799550 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.799728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799579 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:35:33.799728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799619 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:35:33.799861 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799725 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.799861 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.799824 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bgfdp\"" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800031 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800071 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fd02f8mhdkfnh\"" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800195 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800204 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800207 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800338 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.800399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800343 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\"" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800414 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800474 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\"" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800477 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800631 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800725 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.800810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800781 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-grpc-tls\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.801052 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800811 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.801052 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800873 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.801052 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.800949 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.801184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.801104 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.802831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802515 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-federate-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.802932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802893 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.802967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802944 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.803011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802970 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-tls\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.803011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802944 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.803078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803058 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.803112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803072 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.803112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803086 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.803112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803097 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-tls\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.803250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803163 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e300e576-627d-46ea-a62d-567ec3f8d5c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.803250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803243 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98c38cf2-9e21-4d7f-836b-9983359012fc-config-volume\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.803507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.802958 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-client-tls\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803948 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-client-ca-bundle\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803983 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.803985 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804018 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804035 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7977864-d9ee-43d9-9c44-6611a31dba60-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804077 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gsp7d\"" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804104 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98c38cf2-9e21-4d7f-836b-9983359012fc-metrics-tls\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804112 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804123 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804165 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99ab51f7-f13e-41df-985a-172ac7fe4927-secret-metrics-server-client-certs\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804204 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804282 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.804315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804302 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.804800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804476 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99ab51f7-f13e-41df-985a-172ac7fe4927-metrics-server-audit-profiles\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.804800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804629 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.804800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.804672 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.805739 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.805721 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.805833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.805722 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8bf\" (UniqueName: \"kubernetes.io/projected/e7977864-d9ee-43d9-9c44-6611a31dba60-kube-api-access-mm8bf\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808233 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59jq\" (UniqueName: \"kubernetes.io/projected/c58f03ba-4840-4237-88ee-82bf233ac315-kube-api-access-v59jq\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808236 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmbd\" (UniqueName: \"kubernetes.io/projected/fad1332e-70fd-49ce-b1c2-48e50110f29b-kube-api-access-rsmbd\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808468 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7977864-d9ee-43d9-9c44-6611a31dba60-metrics-client-ca\") pod \"thanos-querier-747576895b-zqwt6\" (UID: \"e7977864-d9ee-43d9-9c44-6611a31dba60\") " pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808564 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb29l\" (UniqueName: \"kubernetes.io/projected/140004b8-1074-4ef5-a31e-9c9f52c46d8a-kube-api-access-qb29l\") pod \"ingress-canary-p24hl\" (UID: \"140004b8-1074-4ef5-a31e-9c9f52c46d8a\") " pod="openshift-ingress-canary/ingress-canary-p24hl" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808578 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvzt\" (UniqueName: \"kubernetes.io/projected/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-kube-api-access-6jvzt\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.808789 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.808817 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.808835 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:33.808943 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:34.30891819 +0000 UTC m=+1.758410923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.808945 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7gx\" (UniqueName: \"kubernetes.io/projected/a0797257-801e-4dac-b7ea-ec05f4a382f3-kube-api-access-5z7gx\") pod \"network-metrics-daemon-8lk6r\" (UID: \"a0797257-801e-4dac-b7ea-ec05f4a382f3\") " pod="openshift-multus/network-metrics-daemon-8lk6r" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809027 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ecb26d-be82-4455-8dae-e0b7489350ee-metrics-client-ca\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809312 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809595 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrf6\" (UniqueName: \"kubernetes.io/projected/98c38cf2-9e21-4d7f-836b-9983359012fc-kube-api-access-nbrf6\") pod \"dns-default-pgmdj\" (UID: \"98c38cf2-9e21-4d7f-836b-9983359012fc\") " pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809660 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kqf\" (UniqueName: \"kubernetes.io/projected/d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7-kube-api-access-42kqf\") pod \"monitoring-plugin-77f4f76594-dvj8z\" (UID: \"d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809761 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqfj\" (UniqueName: \"kubernetes.io/projected/6f0c8ec3-e700-43b5-a948-ded642dc21fe-kube-api-access-9vqfj\") pod \"network-check-target-xspfd\" (UID: \"6f0c8ec3-e700-43b5-a948-ded642dc21fe\") " pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809857 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h987g\" (UniqueName: \"kubernetes.io/projected/724b9880-4314-41ad-93f7-40eb026a0fb2-kube-api-access-h987g\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.809900 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810101 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810449 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810483 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n857\" (UniqueName: \"kubernetes.io/projected/d5ecb26d-be82-4455-8dae-e0b7489350ee-kube-api-access-2n857\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810481 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810521 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810535 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810553 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh6l\" (UniqueName: \"kubernetes.io/projected/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-kube-api-access-pdh6l\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810585 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72j9m\" (UniqueName: \"kubernetes.io/projected/e300e576-627d-46ea-a62d-567ec3f8d5c3-kube-api-access-72j9m\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.810761 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/724b9880-4314-41ad-93f7-40eb026a0fb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t5rks\" (UID: \"724b9880-4314-41ad-93f7-40eb026a0fb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t5rks" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811013 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jtg\" (UniqueName: \"kubernetes.io/projected/dc76c3f5-d680-4468-a8ae-b100e41e253c-kube-api-access-t2jtg\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811112 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg285\" (UniqueName: \"kubernetes.io/projected/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-kube-api-access-wg285\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811163 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49cj\" (UniqueName: \"kubernetes.io/projected/814f2828-2c09-40ea-b910-02df6029fffd-kube-api-access-b49cj\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811245 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88zq\" (UniqueName: \"kubernetes.io/projected/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-kube-api-access-d88zq\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811261 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wvp\" (UniqueName: \"kubernetes.io/projected/99ab51f7-f13e-41df-985a-172ac7fe4927-kube-api-access-66wvp\") pod \"metrics-server-695b5bb69b-wlqqm\" (UID: \"99ab51f7-f13e-41df-985a-172ac7fe4927\") " pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811295 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6m25j\" (UID: \"ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6m25j" Oct 30 19:35:33.811324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811349 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw4f\" (UniqueName: \"kubernetes.io/projected/4009f08d-95f6-4e73-ba56-2740e4e55336-kube-api-access-fvw4f\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.812345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811541 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:33.812345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.811597 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z8q\" (UniqueName: \"kubernetes.io/projected/908aaccc-1937-4ac9-8614-28365a9fb984-kube-api-access-t4z8q\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.813620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.813601 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-metrics-client-ca\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.814348 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.814329 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.814458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.814441 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:33.820022 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.819993 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc76c3f5-d680-4468-a8ae-b100e41e253c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84748fcdfb-kqjdr\" (UID: \"dc76c3f5-d680-4468-a8ae-b100e41e253c\") " pod="openshift-monitoring/telemeter-client-84748fcdfb-kqjdr" Oct 30 19:35:33.884968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.884875 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.884968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.884922 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.884968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.884949 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.884980 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-bin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.884994 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-slash\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885003 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-k8s-cni-cncf-io\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885026 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885062 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885080 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-systemd\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885081 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885118 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-multus-certs\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885120 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885147 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885156 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885171 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885190 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885217 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-log-socket\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885206 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885249 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-host\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885256 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-cnibin\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885265 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-registration-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885276 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885296 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885300 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-cnibin\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885310 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885334 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-systemd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885358 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885358 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-kubelet\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885406 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885430 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885433 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-node-exporter-wtmp\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885429 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-run-ovn\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885480 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885489 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-run-netns\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885481 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-system-cni-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885449 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-kubernetes\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885537 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885563 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885617 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885641 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-system-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885644 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-conf-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885663 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885697 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885718 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885737 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885746 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-hostroot\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.885732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885739 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e300e576-627d-46ea-a62d-567ec3f8d5c3-crio-socket\") pod \"insights-runtime-extractor-ts96r\" (UID: \"e300e576-627d-46ea-a62d-567ec3f8d5c3\") " pod="openshift-insights/insights-runtime-extractor-ts96r" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885759 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-etc-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885760 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885799 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885827 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885848 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysconfig\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885854 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885883 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885888 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-socket-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885915 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-modprobe-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885921 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885941 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-cni-multus\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885922 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-var-lib-kubelet\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885947 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885976 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.885948 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-var-lib-openvswitch\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886001 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886023 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886042 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-sys-fs\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886029 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886065 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-netns\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886079 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886106 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886120 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886085 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886144 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886171 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886186 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-cni-dir\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886189 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ac523a2-773b-493f-bb76-8faa6f8c3bb1-host\") pod \"node-ca-ftqg8\" (UID: \"3ac523a2-773b-493f-bb76-8faa6f8c3bb1\") " pod="openshift-image-registry/node-ca-ftqg8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886193 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886208 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886211 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-root\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886172 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557d0d40-9bfa-4d88-9ee2-ac07635e7daa-host-slash\") pod \"iptables-alerter-twm5d\" (UID: \"557d0d40-9bfa-4d88-9ee2-ac07635e7daa\") " pod="openshift-network-operator/iptables-alerter-twm5d" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886192 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-systemd-units\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886244 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886280 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886297 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886319 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886279 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-bin\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886332 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886347 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886359 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-host-cni-netd\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886380 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886383 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-kubelet-config\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886391 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ecb26d-be82-4455-8dae-e0b7489350ee-sys\") pod \"node-exporter-csl8b\" (UID: \"d5ecb26d-be82-4455-8dae-e0b7489350ee\") " pod="openshift-monitoring/node-exporter-csl8b" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886409 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908aaccc-1937-4ac9-8614-28365a9fb984-node-log\") pod \"ovnkube-node-k5fx8\" (UID: \"908aaccc-1937-4ac9-8614-28365a9fb984\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886422 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-run\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886463 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-conf\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886456 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886501 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/814f2828-2c09-40ea-b910-02df6029fffd-device-dir\") pod \"aws-ebs-csi-driver-node-qr9ws\" (UID: \"814f2828-2c09-40ea-b910-02df6029fffd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr9ws" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886500 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886523 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-host-var-lib-kubelet\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886528 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886551 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886567 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wksc9\" (UID: \"3c5f60ac-4e68-4bb9-bd1f-d08bfbc20488\") " pod="openshift-multus/multus-additional-cni-plugins-wksc9" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886575 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886590 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886600 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c58f03ba-4840-4237-88ee-82bf233ac315-hosts-file\") pod \"node-resolver-hmsqs\" (UID: \"c58f03ba-4840-4237-88ee-82bf233ac315\") " pod="openshift-dns/node-resolver-hmsqs" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886612 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886634 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-multus-socket-dir-parent\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886636 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886667 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886675 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad1332e-70fd-49ce-b1c2-48e50110f29b-etc-kubernetes\") pod \"multus-t5d7p\" (UID: \"fad1332e-70fd-49ce-b1c2-48e50110f29b\") " pod="openshift-multus/multus-t5d7p" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886685 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-sys\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886675 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a56f5b-251a-4f17-9dc8-ed6434db1328-dbus\") pod \"global-pull-secret-syncer-zzgh7\" (UID: \"92a56f5b-251a-4f17-9dc8-ed6434db1328\") " pod="kube-system/global-pull-secret-syncer-zzgh7" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886690 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886783 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-lib-modules\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:33.887409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:33.886810 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009f08d-95f6-4e73-ba56-2740e4e55336-etc-sysctl-d\") pod \"tuned-s9qdt\" (UID: \"4009f08d-95f6-4e73-ba56-2740e4e55336\") " pod="openshift-cluster-node-tuning-operator/tuned-s9qdt" Oct 30 19:35:34.057709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.057680 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" Oct 30 19:35:34.291274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.291230 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.291295 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.291336 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.291353 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291358 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.291404 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291450 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.291431945 +0000 UTC m=+2.740924666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291453 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291479 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:34.291491 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291485 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:34.291845 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291532 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.291515532 +0000 UTC m=+2.741008260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:34.291845 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291556 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.291541338 +0000 UTC m=+2.741034048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:34.291845 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291576 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.291566351 +0000 UTC m=+2.741059054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:34.291845 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291491 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:34.291845 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.291632 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.291620334 +0000 UTC m=+2.741113041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:34.392884 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.392839 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:34.393065 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.393019 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:34.393065 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.393047 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:34.393065 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.393060 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:34.393238 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:34.393129 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:35.393110383 +0000 UTC m=+2.842603101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:34.441012 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.440967 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:35:34.605874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:34.605791 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:35.302242 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.302199 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.302305 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.302336 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302349 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.302406 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302427 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.302434 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302489 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.302471435 +0000 UTC m=+4.751964157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302507 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.302498488 +0000 UTC m=+4.751991194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302519 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302521 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302545 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302557 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.302544284 +0000 UTC m=+4.752036988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302579 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.302563033 +0000 UTC m=+4.752055736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:35.302724 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.302599 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.302589912 +0000 UTC m=+4.752082621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:35.403580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.403541 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:35.403776 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.403715 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:35.403776 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.403738 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:35.403776 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.403749 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:35.403879 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:35.403803 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:37.4037901 +0000 UTC m=+4.853282807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:35.695950 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:35.695869 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgmdj" Oct 30 19:35:36.470437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:36.470408 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dvj8z" Oct 30 19:35:37.322882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.322835 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.322882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.322890 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.322925 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.322941 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.322966 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.322969 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323057 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.323035269 +0000 UTC m=+8.772527987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323058 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323066 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323089 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323095 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.323087082 +0000 UTC m=+8.772579808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323065 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323163 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.323146473 +0000 UTC m=+8.772639177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:37.323205 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323207 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.323197207 +0000 UTC m=+8.772689912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:37.323637 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.323220 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.323213039 +0000 UTC m=+8.772705742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:37.424428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:37.424359 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:37.424687 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.424555 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:37.424687 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.424584 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:37.424687 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.424598 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:37.424687 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:37.424668 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:41.424647464 +0000 UTC m=+8.874140168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:38.520114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:38.520084 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:41.369412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.369345 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.369429 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.369462 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.369480 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369483 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.369505 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369537 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369568 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.369547901 +0000 UTC m=+16.819040609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369588 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.369579692 +0000 UTC m=+16.819072395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369595 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369605 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369610 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369646 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.369634039 +0000 UTC m=+16.819126781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369661 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.369654607 +0000 UTC m=+16.819147310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:41.369866 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.369672 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.369666122 +0000 UTC m=+16.819158825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:41.470755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:41.470713 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:41.470964 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.470904 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:41.470964 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.470930 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:41.470964 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.470941 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:41.471123 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:41.471013 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:35:49.470992387 +0000 UTC m=+16.920485111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:43.135357 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:43.135317 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:43.135767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:43.135605 5991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:35:43.645274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:43.645244 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-695b5bb69b-wlqqm" Oct 30 19:35:43.790146 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:43.790100 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5fx8" Oct 30 19:35:49.186097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186062 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:49.186596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186388 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="alertmanager" containerID="cri-o://088c6cca94acbbddf2e0a0e75e8cc8298bff2b28864dddb7f9dadda0e44e3cca" gracePeriod=120 Oct 30 19:35:49.186596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186448 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-metric" containerID="cri-o://b8d898edb5f54bdc1ad25f2217bfda9bfbe69c766dcce8818c758cb593cedff4" gracePeriod=120 Oct 30 19:35:49.186596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186472 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-web" containerID="cri-o://c2ca2eb7fdd7c0c706aa97c787e38e379d090390fd3434c66e056e4531846a7b" gracePeriod=120 Oct 30 19:35:49.186908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186847 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="config-reloader" containerID="cri-o://67cd90bd5cb78bdd7bca02acdb70ec730434ffcd8ed41a4e8d5cbf15a1fb90a2" gracePeriod=120 Oct 30 19:35:49.187068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186495 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy" containerID="cri-o://5820904f4fa174260a2a17648db8dc8c373a39c062f8ae9aa11af6b10d054d7c" gracePeriod=120 Oct 30 19:35:49.187068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.186505 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="prom-label-proxy" containerID="cri-o://412a588f6109afad7e79e4c6ff2be232f441e0635e1e9c30b3da3317a7fdbbcd" gracePeriod=120 Oct 30 19:35:49.459269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.459170 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.459269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.459233 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.459269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.459251 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.459287 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459340 5991 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459390 5991 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.459400 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459412 5991 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459437 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.459416952 +0000 UTC m=+32.908909673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-config" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459448 5991 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459464 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.459454664 +0000 UTC m=+32.908947371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"service-ca" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459464 5991 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459485 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.459476542 +0000 UTC m=+32.908969260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459503 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.459494926 +0000 UTC m=+32.908987629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:35:49.459577 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.459535 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.459518263 +0000 UTC m=+32.909010982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:35:49.560624 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.560578 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"console-7574f7c6c-ksbt6\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:49.560808 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.560749 5991 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:35:49.560808 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.560772 5991 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:35:49.560808 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.560783 5991 projected.go:196] Error preparing data for projected volume kube-api-access-fb57z for pod openshift-console/console-7574f7c6c-ksbt6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:49.560910 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:49.560841 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z podName:4450c8f2-6d00-4a2d-aee2-4acd31855fea nodeName:}" failed. No retries permitted until 2026-10-30 19:36:05.560827779 +0000 UTC m=+33.010320500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb57z" (UniqueName: "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z") pod "console-7574f7c6c-ksbt6" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:35:49.822031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.821995 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="412a588f6109afad7e79e4c6ff2be232f441e0635e1e9c30b3da3317a7fdbbcd" exitCode=0 Oct 30 19:35:49.822031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822022 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="b8d898edb5f54bdc1ad25f2217bfda9bfbe69c766dcce8818c758cb593cedff4" exitCode=0 Oct 30 19:35:49.822031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822028 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="5820904f4fa174260a2a17648db8dc8c373a39c062f8ae9aa11af6b10d054d7c" exitCode=0 Oct 30 19:35:49.822031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822034 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="67cd90bd5cb78bdd7bca02acdb70ec730434ffcd8ed41a4e8d5cbf15a1fb90a2" exitCode=0 Oct 30 19:35:49.822031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822040 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="088c6cca94acbbddf2e0a0e75e8cc8298bff2b28864dddb7f9dadda0e44e3cca" exitCode=0 Oct 30 19:35:49.822451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822070 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"412a588f6109afad7e79e4c6ff2be232f441e0635e1e9c30b3da3317a7fdbbcd"} Oct 30 19:35:49.822451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822185 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"b8d898edb5f54bdc1ad25f2217bfda9bfbe69c766dcce8818c758cb593cedff4"} Oct 30 19:35:49.822451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822225 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"5820904f4fa174260a2a17648db8dc8c373a39c062f8ae9aa11af6b10d054d7c"} Oct 30 19:35:49.822451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822264 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"67cd90bd5cb78bdd7bca02acdb70ec730434ffcd8ed41a4e8d5cbf15a1fb90a2"} Oct 30 19:35:49.822451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:49.822349 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"088c6cca94acbbddf2e0a0e75e8cc8298bff2b28864dddb7f9dadda0e44e3cca"} Oct 30 19:35:50.831087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:50.831054 5991 generic.go:356] "Generic (PLEG): container finished" podID="a4395c20-e68a-4714-99e3-6992284251f6" containerID="c2ca2eb7fdd7c0c706aa97c787e38e379d090390fd3434c66e056e4531846a7b" exitCode=0 Oct 30 19:35:50.831490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:50.831121 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"c2ca2eb7fdd7c0c706aa97c787e38e379d090390fd3434c66e056e4531846a7b"} Oct 30 19:35:50.869107 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:50.869073 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:35:50.945812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:50.945785 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:51.077324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077234 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077283 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077372 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077410 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077436 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077460 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077496 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077530 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077599 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077637 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077672 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077719 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077754 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"a4395c20-e68a-4714-99e3-6992284251f6\" (UID: \"a4395c20-e68a-4714-99e3-6992284251f6\") " Oct 30 19:35:51.077870 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.077760 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:51.078180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.078121 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-metrics-client-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.078180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.078150 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:51.078463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.078436 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:51.082200 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.082152 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.082546 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.082516 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.082702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.082639 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "kube-api-access-nvwws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:51.082841 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.082820 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:51.082909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.082879 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.083109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.083083 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.083478 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.083446 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.083584 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.083490 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:51.087158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.087121 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.095422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.095388 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config" pod "a4395c20-e68a-4714-99e3-6992284251f6" (UID: "a4395c20-e68a-4714-99e3-6992284251f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:51.178882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178845 5991 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-main-db\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.178882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178876 5991 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-cluster-tls-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.178882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178887 5991 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178898 5991 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4395c20-e68a-4714-99e3-6992284251f6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178908 5991 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178917 5991 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-tls-assets\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178926 5991 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-main-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178936 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwws\" (UniqueName: \"kubernetes.io/projected/a4395c20-e68a-4714-99e3-6992284251f6-kube-api-access-nvwws\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178945 5991 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178954 5991 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4395c20-e68a-4714-99e3-6992284251f6-config-out\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178962 5991 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.179114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.178970 5991 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4395c20-e68a-4714-99e3-6992284251f6-web-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:51.837854 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.837817 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4395c20-e68a-4714-99e3-6992284251f6","Type":"ContainerDied","Data":"c6b773be3270eadd161b54913afeda18d82038a08707d3d4eede6b2d8721a945"} Oct 30 19:35:51.837854 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.837850 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:51.838308 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.837904 5991 scope.go:122] "RemoveContainer" containerID="412a588f6109afad7e79e4c6ff2be232f441e0635e1e9c30b3da3317a7fdbbcd" Oct 30 19:35:51.847094 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.847072 5991 scope.go:122] "RemoveContainer" containerID="b8d898edb5f54bdc1ad25f2217bfda9bfbe69c766dcce8818c758cb593cedff4" Oct 30 19:35:51.854970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.854950 5991 scope.go:122] "RemoveContainer" containerID="5820904f4fa174260a2a17648db8dc8c373a39c062f8ae9aa11af6b10d054d7c" Oct 30 19:35:51.863173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.863151 5991 scope.go:122] "RemoveContainer" containerID="c2ca2eb7fdd7c0c706aa97c787e38e379d090390fd3434c66e056e4531846a7b" Oct 30 19:35:51.870781 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.870753 5991 scope.go:122] "RemoveContainer" containerID="67cd90bd5cb78bdd7bca02acdb70ec730434ffcd8ed41a4e8d5cbf15a1fb90a2" Oct 30 19:35:51.878731 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.878703 5991 scope.go:122] "RemoveContainer" containerID="088c6cca94acbbddf2e0a0e75e8cc8298bff2b28864dddb7f9dadda0e44e3cca" Oct 30 19:35:51.886498 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.886480 5991 scope.go:122] "RemoveContainer" containerID="b13258187328279b1f5d16c20293e4f174f3583fe53b66cbc8c729031fc78ee3" Oct 30 19:35:51.899084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.899055 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:51.909141 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.909113 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:51.945783 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.945746 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:51.946055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946044 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946056 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946073 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="config-reloader" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946078 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="config-reloader" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946087 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="alertmanager" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946093 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="alertmanager" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946099 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="init-config-reloader" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946104 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="init-config-reloader" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946114 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-web" Oct 30 19:35:51.946119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946119 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-web" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946128 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="prom-label-proxy" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946132 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="prom-label-proxy" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946141 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-metric" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946147 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-metric" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946197 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-web" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946205 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="prom-label-proxy" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946212 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy-metric" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946220 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="kube-rbac-proxy" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946229 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="config-reloader" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946235 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="alertmanager" Oct 30 19:35:51.946384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.946240 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4395c20-e68a-4714-99e3-6992284251f6" containerName="init-config-reloader" Oct 30 19:35:51.949889 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.949866 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:51.974210 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.974158 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:35:51.978828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.978798 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:51.981235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981197 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:35:51.981235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981214 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:35:51.981235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981229 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:35:51.981468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981267 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:35:51.981468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981274 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:35:51.981468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981330 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:35:51.981468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981335 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:35:51.981468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981406 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p99nr\"" Oct 30 19:35:51.981632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:51.981488 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:35:52.088241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088141 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frpc\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-kube-api-access-4frpc\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088197 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088281 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088347 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-config-volume\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088484 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088511 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088543 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088560 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-web-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088692 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088732 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-config-out\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088845 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088783 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088839 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.088887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.088866 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189479 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189535 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-web-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189577 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189603 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-config-out\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189623 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.189800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189670 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189792 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189866 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frpc\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-kube-api-access-4frpc\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189905 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189938 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.189974 5991 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:35:52.190245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.190032 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-config-volume\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.190148 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.190195 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190495 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.190473 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.190896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.190701 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.191204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.191177 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df553a58-0c71-4068-9409-f67d91ad6150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.192982 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.192984 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-web-config\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193234 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193154 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193290 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193260 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193347 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-config-volume\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193358 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193588 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193570 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.193675 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.193659 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df553a58-0c71-4068-9409-f67d91ad6150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.194710 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.194686 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df553a58-0c71-4068-9409-f67d91ad6150-config-out\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.208954 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.208927 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frpc\" (UniqueName: \"kubernetes.io/projected/df553a58-0c71-4068-9409-f67d91ad6150-kube-api-access-4frpc\") pod \"alertmanager-main-0\" (UID: \"df553a58-0c71-4068-9409-f67d91ad6150\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.268513 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.268473 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:35:52.592224 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.592160 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:35:52.595617 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:52.595585 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf553a58_0c71_4068_9409_f67d91ad6150.slice/crio-18c0cb1aee3efb30558fe1c55250a7f0b7b7df3cf5c202ecc488a4ce199ba17d WatchSource:0}: Error finding container 18c0cb1aee3efb30558fe1c55250a7f0b7b7df3cf5c202ecc488a4ce199ba17d: Status 404 returned error can't find the container with id 18c0cb1aee3efb30558fe1c55250a7f0b7b7df3cf5c202ecc488a4ce199ba17d Oct 30 19:35:52.681467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.681433 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4395c20-e68a-4714-99e3-6992284251f6" path="/var/lib/kubelet/pods/a4395c20-e68a-4714-99e3-6992284251f6/volumes" Oct 30 19:35:52.842885 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.842798 5991 generic.go:356] "Generic (PLEG): container finished" podID="df553a58-0c71-4068-9409-f67d91ad6150" containerID="af52d0d3b17b069776e22a297b27ebbe0ba41851d29189e8d2227ce47de8c91a" exitCode=0 Oct 30 19:35:52.843438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.842894 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerDied","Data":"af52d0d3b17b069776e22a297b27ebbe0ba41851d29189e8d2227ce47de8c91a"} Oct 30 19:35:52.843438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:52.842968 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"18c0cb1aee3efb30558fe1c55250a7f0b7b7df3cf5c202ecc488a4ce199ba17d"} Oct 30 19:35:53.827470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827387 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:53.827713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827682 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="prometheus" containerID="cri-o://d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" gracePeriod=600 Oct 30 19:35:53.827713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827700 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy" containerID="cri-o://a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" gracePeriod=600 Oct 30 19:35:53.827907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827712 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="thanos-sidecar" containerID="cri-o://b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" gracePeriod=600 Oct 30 19:35:53.827907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827740 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="config-reloader" containerID="cri-o://cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" gracePeriod=600 Oct 30 19:35:53.827907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827773 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" gracePeriod=600 Oct 30 19:35:53.827907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.827721 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-web" containerID="cri-o://4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" gracePeriod=600 Oct 30 19:35:53.852343 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852315 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"f2015e1aa9be111179ed859a9d4e1e6d201aa9a42b68d52e36e94e6d9dee0d6b"} Oct 30 19:35:53.853006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852396 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"6161a09cdecdeca71ef309ed7c46f19521c82e33e34b15ba9b7490c2f564aa15"} Oct 30 19:35:53.853006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852429 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"29234a6be709243de3257c966bc355731703ff5dcf51d9d9dc46fcbb806e19ba"} Oct 30 19:35:53.853006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852462 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"12573f850920558bf8021a21894c887a8051a824ad952509c0597e4f1f5ba915"} Oct 30 19:35:53.853006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852485 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"cab606c5d884646d7729a2367ba023a410e93d98f3f5df4c3729ed819733c922"} Oct 30 19:35:53.853006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.852519 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df553a58-0c71-4068-9409-f67d91ad6150","Type":"ContainerStarted","Data":"036e3992f5150f05ab54c5cf98f271ab5c5cdf8784be6d780926aafb25707bb2"} Oct 30 19:35:53.942312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:53.942257 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9268916689999998 podStartE2EDuration="2.942244362s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.352693ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:53.942019141 +0000 UTC m=+21.391511893" watchObservedRunningTime="2026-10-30 19:35:53.942244362 +0000 UTC m=+21.391737081" Oct 30 19:35:54.077540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.077477 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:54.211727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211692 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211743 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211773 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211804 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211840 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211886 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.211915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.211913 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212012 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212060 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212096 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212141 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212175 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212243 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212278 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212307 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212343 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212389 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212407 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212439 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212487 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.212806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212492 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.212806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212501 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.212806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212522 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") pod \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\" (UID: \"b5533697-e0c7-42b4-958f-cbfc3541bc3e\") " Oct 30 19:35:54.213030 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212889 5991 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.213030 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212911 5991 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.213030 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.212927 5991 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.213245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.213213 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.214159 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.213760 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:54.214656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.214628 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.214762 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.214669 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.215484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.215315 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.215484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.215425 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.215730 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.215699 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.215887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.215861 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.216610 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.216583 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:54.216610 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.216602 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.216792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.216729 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:54.216988 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.216958 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.217417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.217394 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.217516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.217494 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "kube-api-access-59zt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:54.217604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.217589 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.218090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.218073 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.227355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.227321 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config" pod "b5533697-e0c7-42b4-958f-cbfc3541bc3e" (UID: "b5533697-e0c7-42b4-958f-cbfc3541bc3e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.313974 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.313939 5991 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.313974 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.313969 5991 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-kube-rbac-proxy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.313974 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.313981 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59zt2\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-kube-api-access-59zt2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.313990 5991 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.313999 5991 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314008 5991 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-grpc-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314017 5991 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314026 5991 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5533697-e0c7-42b4-958f-cbfc3541bc3e-tls-assets\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314034 5991 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-web-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314043 5991 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-configmap-metrics-client-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314052 5991 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314061 5991 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-db\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314069 5991 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-metrics-client-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314077 5991 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5533697-e0c7-42b4-958f-cbfc3541bc3e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314086 5991 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314094 5991 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5533697-e0c7-42b4-958f-cbfc3541bc3e-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.314102 5991 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5533697-e0c7-42b4-958f-cbfc3541bc3e-config-out\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:54.860405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860374 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" exitCode=0 Oct 30 19:35:54.860405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860400 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" exitCode=0 Oct 30 19:35:54.860405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860409 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" exitCode=0 Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860415 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" exitCode=0 Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860421 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" exitCode=0 Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860427 5991 generic.go:356] "Generic (PLEG): container finished" podID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" exitCode=0 Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860457 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860456 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860591 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860621 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860645 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860654 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860669 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860694 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} Oct 30 19:35:54.860893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.860725 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5533697-e0c7-42b4-958f-cbfc3541bc3e","Type":"ContainerDied","Data":"fb973f7e4a5e2f10fd8459170b79aeb67829175017949b1c4a4a25b7366ebc1c"} Oct 30 19:35:54.868418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.868391 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.876064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.876045 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.883249 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.883223 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.890543 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.890520 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.897982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.897959 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.905853 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.905832 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.913728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.913703 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.914070 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.914048 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.914115 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914081 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.914159 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914117 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.914391 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.914355 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.914468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914402 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.914468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914425 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.914723 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.914706 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.914760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914729 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.914760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.914743 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.914994 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.914976 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.915039 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915001 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.915039 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915016 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.915250 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.915233 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.915312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915257 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.915312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915275 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.915563 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.915543 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.915600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915579 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.915640 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915597 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.915861 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:35:54.915821 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.915903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915867 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.915903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.915882 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.916152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916124 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.916192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916152 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.916389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916358 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.916389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916388 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.916603 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916587 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.916655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916610 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.916853 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916833 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.916898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.916853 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.917087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917065 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.917132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917089 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.917299 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917280 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.917299 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917298 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.917504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917486 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.917504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917503 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.917684 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917665 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.917724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917684 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.917858 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917839 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.917912 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.917859 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.918041 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918021 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.918088 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918042 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.918257 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918238 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.918339 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918259 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.918484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918467 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.918528 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918483 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.918686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918668 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.918730 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918686 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.918945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918928 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.918992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.918946 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.919143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919126 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.919181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919142 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.919377 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919335 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.919377 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919358 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.919591 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919572 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.919640 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919592 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.919796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919773 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.919860 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.919796 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.920064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920033 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.920158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920063 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.920384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920340 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.920469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920386 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.920672 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920653 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.920712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920673 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.920929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920912 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.920929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.920928 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.921149 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921130 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.921149 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921146 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.921340 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921320 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.921415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921340 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.921556 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921538 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.921625 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921556 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.921748 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921729 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.921793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921748 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.921972 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921956 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.921972 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.921971 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.922162 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922147 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.922208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922163 5991 scope.go:122] "RemoveContainer" containerID="5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba" Oct 30 19:35:54.922354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922335 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba"} err="failed to get container status \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": rpc error: code = NotFound desc = could not find container \"5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba\": container with ID starting with 5a153d13f875c24d902649fd7efe9c66277a9be9ce34ff36411964931f0f42ba not found: ID does not exist" Oct 30 19:35:54.922412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922353 5991 scope.go:122] "RemoveContainer" containerID="a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9" Oct 30 19:35:54.922604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922586 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9"} err="failed to get container status \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": rpc error: code = NotFound desc = could not find container \"a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9\": container with ID starting with a4c01d6f157a7e4d3f67e45330c3c81d48492de1130bcd9877a4f80594be0cb9 not found: ID does not exist" Oct 30 19:35:54.922604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922603 5991 scope.go:122] "RemoveContainer" containerID="4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642" Oct 30 19:35:54.922823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922808 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642"} err="failed to get container status \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": rpc error: code = NotFound desc = could not find container \"4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642\": container with ID starting with 4ec55c025f4ee86350e145ad34a03eb2dfea220973a26bb03b7c353801f50642 not found: ID does not exist" Oct 30 19:35:54.922823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.922824 5991 scope.go:122] "RemoveContainer" containerID="b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc" Oct 30 19:35:54.923047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923031 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc"} err="failed to get container status \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": rpc error: code = NotFound desc = could not find container \"b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc\": container with ID starting with b35dd7ba33ecf24a26358b823ed038c18e098661bc0159171d8d81098a384bfc not found: ID does not exist" Oct 30 19:35:54.923047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923046 5991 scope.go:122] "RemoveContainer" containerID="cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607" Oct 30 19:35:54.923265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923245 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607"} err="failed to get container status \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": rpc error: code = NotFound desc = could not find container \"cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607\": container with ID starting with cd9085bfb3f7ad774fb4527547267780a4c22f9d7ccac52d3f15b5b7b174d607 not found: ID does not exist" Oct 30 19:35:54.923265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923265 5991 scope.go:122] "RemoveContainer" containerID="d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2" Oct 30 19:35:54.923494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923478 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2"} err="failed to get container status \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": rpc error: code = NotFound desc = could not find container \"d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2\": container with ID starting with d3c6a308a0b61769c53e2d9c2fd9603d6c85e68abc5316a424436e9f0db11bc2 not found: ID does not exist" Oct 30 19:35:54.923538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923494 5991 scope.go:122] "RemoveContainer" containerID="f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25" Oct 30 19:35:54.923690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.923674 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25"} err="failed to get container status \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": rpc error: code = NotFound desc = could not find container \"f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25\": container with ID starting with f2efeb894fb58d065364b77cc053eb5c04ffc96a60bffb5334e8beacaa45ac25 not found: ID does not exist" Oct 30 19:35:54.928512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.928487 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:54.939289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.939205 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:54.965882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.965847 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:54.966180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966165 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-thanos" Oct 30 19:35:54.966222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966184 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-thanos" Oct 30 19:35:54.966222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966204 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="init-config-reloader" Oct 30 19:35:54.966222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966209 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="init-config-reloader" Oct 30 19:35:54.966222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966217 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="thanos-sidecar" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966225 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="thanos-sidecar" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966234 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-web" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966239 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-web" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966245 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="config-reloader" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966251 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="config-reloader" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966279 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966284 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966291 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="prometheus" Oct 30 19:35:54.966338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966295 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="prometheus" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966373 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-web" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966383 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="config-reloader" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966394 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966401 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="prometheus" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966408 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="kube-rbac-proxy-thanos" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966419 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="init-config-reloader" Oct 30 19:35:54.966602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.966427 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" containerName="thanos-sidecar" Oct 30 19:35:54.972047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.972021 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:54.985518 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985491 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:35:54.985700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985552 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:35:54.985700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985574 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:35:54.985700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985609 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:35:54.985700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985672 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:35:54.985889 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985821 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:35:54.985889 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985831 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:35:54.985889 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.985821 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:35:54.986076 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986058 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgcwn\"" Oct 30 19:35:54.986295 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986082 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:35:54.986295 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986094 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9e0lqdo2eio9\"" Oct 30 19:35:54.986295 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986118 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:35:54.986427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986408 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:35:54.986470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.986440 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:35:54.996976 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.996913 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:35:54.997408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.997391 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:35:54.999674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:54.999473 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:55.122150 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122101 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122304 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122173 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122304 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122260 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122304 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122294 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122315 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122394 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122452 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rx8\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-kube-api-access-g5rx8\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122565 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122499 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122595 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122566 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122595 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122583 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122649 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122601 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122649 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122620 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122674 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122696 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122766 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122753 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122799 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122785 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122860 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122832 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122887 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122925 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122905 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.122925 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.122921 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223593 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223633 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223671 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223691 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223706 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223730 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.223788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223759 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223803 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223844 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223873 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223896 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223917 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rx8\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-kube-api-access-g5rx8\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223961 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.223994 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224016 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224043 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224069 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224100 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224125 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224167 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224308 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.224593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224567 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.225766 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.224880 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.225766 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.225350 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.225766 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.225438 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.226749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.226634 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.226938 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.226893 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.227154 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.227130 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.227288 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.227265 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.227392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.227284 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.227392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.227302 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.227861 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.227838 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.228059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.228014 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.228184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.228164 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.229147 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.229126 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.229798 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.229777 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.229901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.229830 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.230048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.230030 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.231410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.231388 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.243437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.243402 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rx8\" (UniqueName: \"kubernetes.io/projected/a6a8e960-a61f-4dfa-9ad7-dbc6c7442873-kube-api-access-g5rx8\") pod \"prometheus-k8s-0\" (UID: \"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.297691 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.297654 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:35:55.629177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.629103 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:35:55.632993 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:35:55.632959 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a8e960_a61f_4dfa_9ad7_dbc6c7442873.slice/crio-7ef26272205e40e02ef28ab32444e0c53d8447f34b05ba8f5a1ef800c54b5cef WatchSource:0}: Error finding container 7ef26272205e40e02ef28ab32444e0c53d8447f34b05ba8f5a1ef800c54b5cef: Status 404 returned error can't find the container with id 7ef26272205e40e02ef28ab32444e0c53d8447f34b05ba8f5a1ef800c54b5cef Oct 30 19:35:55.866080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.866045 5991 generic.go:356] "Generic (PLEG): container finished" podID="a6a8e960-a61f-4dfa-9ad7-dbc6c7442873" containerID="3bf627d5dbd769fcd65e308289b8625bac88659e69032d4e40e33ba5a6fa2feb" exitCode=0 Oct 30 19:35:55.866527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.866132 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerDied","Data":"3bf627d5dbd769fcd65e308289b8625bac88659e69032d4e40e33ba5a6fa2feb"} Oct 30 19:35:55.866527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:55.866214 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"7ef26272205e40e02ef28ab32444e0c53d8447f34b05ba8f5a1ef800c54b5cef"} Oct 30 19:35:56.680192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.678212 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5533697-e0c7-42b4-958f-cbfc3541bc3e" path="/var/lib/kubelet/pods/b5533697-e0c7-42b4-958f-cbfc3541bc3e/volumes" Oct 30 19:35:56.879161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879126 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"737cb4ead3115089313e69423a7d08a5889c875f8556ac09f27f50b0ae609114"} Oct 30 19:35:56.879568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879194 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"dfc06f58b3fbe1d615a78c3b5e677bb18b8b7a717e833465546e7c9fcba7df8d"} Oct 30 19:35:56.879568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879220 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"d1413bb7822c536700f67598a7b931d3680eac308b7596e3ab20475fa81ccfaf"} Oct 30 19:35:56.879568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879244 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"9bc15e589cad478f25267a2401811e6f7dbf4715acb42b60c27cdc8e77671974"} Oct 30 19:35:56.879568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879267 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"fcfe34ac75e79dc79e3dd53951b68adae1c47696fe8997abc675e33fa779711b"} Oct 30 19:35:56.879568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.879288 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6a8e960-a61f-4dfa-9ad7-dbc6c7442873","Type":"ContainerStarted","Data":"30a2f1866228226f8369f1934ca83d2428e686aa9f778af25e0a2f0de1f2a2c5"} Oct 30 19:35:56.937485 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:56.937386 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9222373619999997 podStartE2EDuration="2.937354139s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.116777ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:56.936053699 +0000 UTC m=+24.385546451" watchObservedRunningTime="2026-10-30 19:35:56.937354139 +0000 UTC m=+24.386846864" Oct 30 19:35:58.707246 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:58.707192 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7574f7c6c-ksbt6" podUID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" containerName="console" containerID="cri-o://665f4aed7c2266d46ec120cdbd27593e0f9ad4fa17c32438cfbc87a3aae700ac" gracePeriod=15 Oct 30 19:35:58.890329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:58.890298 5991 generic.go:356] "Generic (PLEG): container finished" podID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" containerID="665f4aed7c2266d46ec120cdbd27593e0f9ad4fa17c32438cfbc87a3aae700ac" exitCode=0 Oct 30 19:35:58.890522 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:58.890384 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574f7c6c-ksbt6" event={"ID":"4450c8f2-6d00-4a2d-aee2-4acd31855fea","Type":"ContainerDied","Data":"665f4aed7c2266d46ec120cdbd27593e0f9ad4fa17c32438cfbc87a3aae700ac"} Oct 30 19:35:58.952506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:58.952479 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:59.064957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.064920 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.064957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.064965 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.064986 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065043 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065072 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065111 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065233 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") pod \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\" (UID: \"4450c8f2-6d00-4a2d-aee2-4acd31855fea\") " Oct 30 19:35:59.065507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065475 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:59.065570 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065505 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:59.065570 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065511 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:59.065570 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065544 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:59.065717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065618 5991 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4450c8f2-6d00-4a2d-aee2-4acd31855fea-tmp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.065717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065632 5991 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.065717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065641 5991 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-oauth-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.065717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.065649 5991 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.067556 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.067512 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:59.067909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.067570 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:59.067909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.067614 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z" pod "4450c8f2-6d00-4a2d-aee2-4acd31855fea" (UID: "4450c8f2-6d00-4a2d-aee2-4acd31855fea"). InnerVolumeSpecName "kube-api-access-fb57z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:59.166204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.166163 5991 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-oauth-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.166204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.166193 5991 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4450c8f2-6d00-4a2d-aee2-4acd31855fea-console-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.166204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.166203 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb57z\" (UniqueName: \"kubernetes.io/projected/4450c8f2-6d00-4a2d-aee2-4acd31855fea-kube-api-access-fb57z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:35:59.896075 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.896038 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574f7c6c-ksbt6" event={"ID":"4450c8f2-6d00-4a2d-aee2-4acd31855fea","Type":"ContainerDied","Data":"74ce61a1d1aa93fcd8682ba5606b0bf1054959094e0d88023bc2c30f581705aa"} Oct 30 19:35:59.896075 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.896064 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574f7c6c-ksbt6" Oct 30 19:35:59.896536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.896109 5991 scope.go:122] "RemoveContainer" containerID="665f4aed7c2266d46ec120cdbd27593e0f9ad4fa17c32438cfbc87a3aae700ac" Oct 30 19:35:59.939568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.939532 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:35:59.949307 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:35:59.949274 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7574f7c6c-ksbt6"] Oct 30 19:36:00.298513 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:00.298465 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:36:00.675198 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:00.675098 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" path="/var/lib/kubelet/pods/4450c8f2-6d00-4a2d-aee2-4acd31855fea/volumes" Oct 30 19:36:02.364650 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:02.364620 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7lfb" Oct 30 19:36:04.756164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:04.756131 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xspfd" Oct 30 19:36:20.868302 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:36:20.868261 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:36:25.478554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:25.478496 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:36:25.496193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:25.496162 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:36:26.012605 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:36:26.012572 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:36:50.867076 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:36:50.867037 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:37:20.867553 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:37:20.867466 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:37:50.867716 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:37:50.867674 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:38:20.866842 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:38:20.866805 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:38:50.866753 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:38:50.866675 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:39:20.867642 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:39:20.867600 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:39:50.868475 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:39:50.868436 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:40:20.868414 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:40:20.868317 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:40:32.704665 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:40:32.704633 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:40:32.706760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:40:32.706740 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:40:50.867963 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:40:50.867925 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:41:10.163011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.162958 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw"] Oct 30 19:41:10.163521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.163420 5991 cpu_manager.go:414] "containerMap: removing container" podUID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" containerName="console" Oct 30 19:41:10.163521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.163436 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" containerName="console" Oct 30 19:41:10.163521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.163517 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4450c8f2-6d00-4a2d-aee2-4acd31855fea" containerName="console" Oct 30 19:41:10.166510 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.166485 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.178131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.178108 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-tmp\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.178262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.178141 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xrv\" (UniqueName: \"kubernetes.io/projected/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-kube-api-access-j4xrv\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.183495 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.183476 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:41:10.183598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.183508 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:10.193109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.193085 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-w925t\"" Oct 30 19:41:10.193903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.193882 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw"] Oct 30 19:41:10.279095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.279051 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-tmp\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.279095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.279100 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xrv\" (UniqueName: \"kubernetes.io/projected/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-kube-api-access-j4xrv\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.279502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.279481 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-tmp\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.304057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.304029 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xrv\" (UniqueName: \"kubernetes.io/projected/4b2ac6c6-d2cd-4b14-b115-cf77f765fd81-kube-api-access-j4xrv\") pod \"jobset-operator-674cd59dc5-vbzvw\" (UID: \"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.492039 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.492000 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" Oct 30 19:41:10.785486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.785399 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw"] Oct 30 19:41:10.789257 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:10.789230 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b2ac6c6_d2cd_4b14_b115_cf77f765fd81.slice/crio-c0c8fa03bc71d1c91e86ad1512f77553b9c23c97e81e8337b61c18ded4bf1e88 WatchSource:0}: Error finding container c0c8fa03bc71d1c91e86ad1512f77553b9c23c97e81e8337b61c18ded4bf1e88: Status 404 returned error can't find the container with id c0c8fa03bc71d1c91e86ad1512f77553b9c23c97e81e8337b61c18ded4bf1e88 Oct 30 19:41:10.791112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:10.791089 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:41:11.032108 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:11.032070 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" event={"ID":"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81","Type":"ContainerStarted","Data":"c0c8fa03bc71d1c91e86ad1512f77553b9c23c97e81e8337b61c18ded4bf1e88"} Oct 30 19:41:14.047579 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:14.047345 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" event={"ID":"4b2ac6c6-d2cd-4b14-b115-cf77f765fd81","Type":"ContainerStarted","Data":"985bd192888397bc6c2ceaa843a2438d5271ab208b0cc6e0e8efb347bb0479cc"} Oct 30 19:41:14.087978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:14.087923 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vbzvw" podStartSLOduration=1.49019056 podStartE2EDuration="4.087907334s" totalImagesPullingTime="2.597716774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:14.087360223 +0000 UTC m=+341.536852952" watchObservedRunningTime="2026-10-30 19:41:14.087907334 +0000 UTC m=+341.537400052" Oct 30 19:41:19.852237 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.852196 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw"] Oct 30 19:41:19.855848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.855823 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.892265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.892232 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw"] Oct 30 19:41:19.945606 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.945571 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:41:19.945968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.945942 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:41:19.946057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.946007 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:41:19.946193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.946174 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:41:19.946253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.946237 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:41:19.946309 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.946288 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zx96v\"" Oct 30 19:41:19.969567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969528 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.969567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969568 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.969791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969587 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spj7\" (UniqueName: \"kubernetes.io/projected/6b2caad3-91dc-45f2-90ea-44a031d0db86-kube-api-access-8spj7\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.969791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969684 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.969791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969747 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:19.969906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:19.969846 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070548 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spj7\" (UniqueName: \"kubernetes.io/projected/6b2caad3-91dc-45f2-90ea-44a031d0db86-kube-api-access-8spj7\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070587 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070619 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070706 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070730 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.070850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.070754 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.071255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.071172 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.073187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.073156 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2caad3-91dc-45f2-90ea-44a031d0db86-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.073528 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.073508 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.073610 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.073552 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.075044 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.075024 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2caad3-91dc-45f2-90ea-44a031d0db86-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.090403 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.089803 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spj7\" (UniqueName: \"kubernetes.io/projected/6b2caad3-91dc-45f2-90ea-44a031d0db86-kube-api-access-8spj7\") pod \"opendatahub-operator-controller-manager-54879d4f6b-8ppkw\" (UID: \"6b2caad3-91dc-45f2-90ea-44a031d0db86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.171808 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.171715 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:20.588293 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:20.588264 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw"] Oct 30 19:41:20.592129 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:20.592095 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2caad3_91dc_45f2_90ea_44a031d0db86.slice/crio-647851d0a047e4e49931d790c32468e833f264bb38851a3420fa04730b5ea6ab WatchSource:0}: Error finding container 647851d0a047e4e49931d790c32468e833f264bb38851a3420fa04730b5ea6ab: Status 404 returned error can't find the container with id 647851d0a047e4e49931d790c32468e833f264bb38851a3420fa04730b5ea6ab Oct 30 19:41:20.867932 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:41:20.867845 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:41:21.076421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:21.076354 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" event={"ID":"6b2caad3-91dc-45f2-90ea-44a031d0db86","Type":"ContainerStarted","Data":"647851d0a047e4e49931d790c32468e833f264bb38851a3420fa04730b5ea6ab"} Oct 30 19:41:24.092208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:24.092174 5991 generic.go:356] "Generic (PLEG): container finished" podID="6b2caad3-91dc-45f2-90ea-44a031d0db86" containerID="1d0b700ff9f68545549f36a8097d701faaa8af617e1f50ba0d62083b3709c91c" exitCode=0 Oct 30 19:41:24.092642 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:24.092226 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" event={"ID":"6b2caad3-91dc-45f2-90ea-44a031d0db86","Type":"ContainerDied","Data":"1d0b700ff9f68545549f36a8097d701faaa8af617e1f50ba0d62083b3709c91c"} Oct 30 19:41:25.099989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:25.099952 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" event={"ID":"6b2caad3-91dc-45f2-90ea-44a031d0db86","Type":"ContainerStarted","Data":"1fd0f30cb7565113216c5f8a0f1cb101cd4e854c3d99b7ddc0968f223ed87aa5"} Oct 30 19:41:25.100544 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:25.100055 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:25.187011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:25.186925 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" podStartSLOduration=2.528206617 podStartE2EDuration="6.186911571s" totalImagesPullingTime="3.162222021s" totalInitContainerRuntime="496.482933ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:25.176494922 +0000 UTC m=+352.625987647" watchObservedRunningTime="2026-10-30 19:41:25.186911571 +0000 UTC m=+352.636404296" Oct 30 19:41:36.106856 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:36.106827 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-8ppkw" Oct 30 19:41:37.169502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.169467 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-j8486"] Oct 30 19:41:37.262180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.262148 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-j8486"] Oct 30 19:41:37.262320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.262241 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.290689 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.290661 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lj4nr\"" Oct 30 19:41:37.302879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.302852 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:41:37.303029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.302931 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:41:37.425771 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.425697 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.425771 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.425748 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvw6\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-kube-api-access-jfvw6\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.526416 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.526375 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.526578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.526443 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvw6\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-kube-api-access-jfvw6\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.544429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.544400 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.545061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.545037 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvw6\" (UniqueName: \"kubernetes.io/projected/1bac1e18-7114-4164-a803-75f43023a005-kube-api-access-jfvw6\") pod \"cert-manager-webhook-5dbbbf75cc-j8486\" (UID: \"1bac1e18-7114-4164-a803-75f43023a005\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.583580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.583554 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:37.921190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:37.921153 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-j8486"] Oct 30 19:41:37.925434 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:37.925410 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bac1e18_7114_4164_a803_75f43023a005.slice/crio-1af88d7d02e5d65a57cdd18e048ae4c250145e16566e35dfc15d45967a6e650a WatchSource:0}: Error finding container 1af88d7d02e5d65a57cdd18e048ae4c250145e16566e35dfc15d45967a6e650a: Status 404 returned error can't find the container with id 1af88d7d02e5d65a57cdd18e048ae4c250145e16566e35dfc15d45967a6e650a Oct 30 19:41:38.154102 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:38.154066 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" event={"ID":"1bac1e18-7114-4164-a803-75f43023a005","Type":"ContainerStarted","Data":"1af88d7d02e5d65a57cdd18e048ae4c250145e16566e35dfc15d45967a6e650a"} Oct 30 19:41:39.121472 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.119840 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf"] Oct 30 19:41:39.123110 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.122525 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.136273 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.136251 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x78xk\"" Oct 30 19:41:39.150677 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.150622 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf"] Oct 30 19:41:39.241084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.241038 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npwxc\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-kube-api-access-npwxc\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.241297 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.241089 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.341869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.341814 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwxc\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-kube-api-access-npwxc\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.341869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.341854 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.367157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.367122 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.367333 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.367223 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwxc\" (UniqueName: \"kubernetes.io/projected/392975b7-4916-4309-a352-278fb56ecfc0-kube-api-access-npwxc\") pod \"cert-manager-cainjector-5c57bbd4fc-brswf\" (UID: \"392975b7-4916-4309-a352-278fb56ecfc0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.436400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.436305 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" Oct 30 19:41:39.770096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:39.770052 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf"] Oct 30 19:41:39.774782 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:39.774753 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod392975b7_4916_4309_a352_278fb56ecfc0.slice/crio-c42aac60bb7969c2016b89c8028920b6adad9e6aa34c721f74a17e43c9dae35e WatchSource:0}: Error finding container c42aac60bb7969c2016b89c8028920b6adad9e6aa34c721f74a17e43c9dae35e: Status 404 returned error can't find the container with id c42aac60bb7969c2016b89c8028920b6adad9e6aa34c721f74a17e43c9dae35e Oct 30 19:41:40.165986 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:40.165898 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" event={"ID":"392975b7-4916-4309-a352-278fb56ecfc0","Type":"ContainerStarted","Data":"c42aac60bb7969c2016b89c8028920b6adad9e6aa34c721f74a17e43c9dae35e"} Oct 30 19:41:42.177114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:42.177031 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" event={"ID":"1bac1e18-7114-4164-a803-75f43023a005","Type":"ContainerStarted","Data":"7f31d75f8bdd387c076da1ae43c0fbc2699f22460e45fd5341fbdd89ffab64b0"} Oct 30 19:41:42.177542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:42.177166 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:42.238431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:42.238354 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" podStartSLOduration=1.433397429 podStartE2EDuration="5.238340452s" totalImagesPullingTime="3.804943023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:42.237792525 +0000 UTC m=+369.687285257" watchObservedRunningTime="2026-10-30 19:41:42.238340452 +0000 UTC m=+369.687833176" Oct 30 19:41:43.182169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:43.182130 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" event={"ID":"392975b7-4916-4309-a352-278fb56ecfc0","Type":"ContainerStarted","Data":"29a54f629487f673f197ab82be15535e2f0d3e65ed05efe054dc96eb80dc37ce"} Oct 30 19:41:43.232721 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:43.232663 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-brswf" podStartSLOduration=1.496951301 podStartE2EDuration="4.232649418s" totalImagesPullingTime="2.735698117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:43.232091888 +0000 UTC m=+370.681584613" watchObservedRunningTime="2026-10-30 19:41:43.232649418 +0000 UTC m=+370.682142144" Oct 30 19:41:44.685723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.685646 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b"] Oct 30 19:41:44.688934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.688912 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.702765 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.702740 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 19:41:44.712337 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.712317 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-526g2\"" Oct 30 19:41:44.712520 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.712499 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:41:44.712596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.712569 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 19:41:44.712596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.712574 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:41:44.713918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.713897 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b"] Oct 30 19:41:44.796566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.796530 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.796742 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.796572 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvssd\" (UniqueName: \"kubernetes.io/projected/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-kube-api-access-pvssd\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.796797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.796757 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.897729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.897692 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.897902 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.897764 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.897902 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.897793 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvssd\" (UniqueName: \"kubernetes.io/projected/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-kube-api-access-pvssd\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.898569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.898550 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.900161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.900132 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:44.915902 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:44.915873 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvssd\" (UniqueName: \"kubernetes.io/projected/b1bda3e4-bb2b-4269-9ddf-0c536750dfc1-kube-api-access-pvssd\") pod \"kuadrant-console-plugin-66c4fb575d-mg72b\" (UID: \"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:45.000441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:45.000414 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" Oct 30 19:41:45.300139 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:45.300054 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b"] Oct 30 19:41:45.303866 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:45.303840 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1bda3e4_bb2b_4269_9ddf_0c536750dfc1.slice/crio-1ef4a3f06a9ad19ebcdf4006187f220b4e5bdd48886bfe2bdb4f7cac303f7133 WatchSource:0}: Error finding container 1ef4a3f06a9ad19ebcdf4006187f220b4e5bdd48886bfe2bdb4f7cac303f7133: Status 404 returned error can't find the container with id 1ef4a3f06a9ad19ebcdf4006187f220b4e5bdd48886bfe2bdb4f7cac303f7133 Oct 30 19:41:46.195759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:46.195723 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" event={"ID":"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1","Type":"ContainerStarted","Data":"1ef4a3f06a9ad19ebcdf4006187f220b4e5bdd48886bfe2bdb4f7cac303f7133"} Oct 30 19:41:46.936573 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:46.936537 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjxr8"] Oct 30 19:41:46.940176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:46.940154 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:46.964535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:46.964509 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t4pnh\"" Oct 30 19:41:46.965803 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:46.965776 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjxr8"] Oct 30 19:41:47.015089 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.015053 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwmk\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-kube-api-access-6kwmk\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.015264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.015170 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-bound-sa-token\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.115984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.115950 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-bound-sa-token\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.116181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.116036 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwmk\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-kube-api-access-6kwmk\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.135937 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.135912 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-bound-sa-token\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.138113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.138086 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwmk\" (UniqueName: \"kubernetes.io/projected/8cdfc4e4-12ba-463b-aabc-e66376c6fb8d-kube-api-access-6kwmk\") pod \"cert-manager-597b95cb84-bjxr8\" (UID: \"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d\") " pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.255950 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.255049 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bjxr8" Oct 30 19:41:47.558709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:47.558456 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjxr8"] Oct 30 19:41:47.562915 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:47.562873 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cdfc4e4_12ba_463b_aabc_e66376c6fb8d.slice/crio-51c45f7c2f60d3298f8a8081c170b8fc8b23987ef306e67c43764e8dd0dbd94e WatchSource:0}: Error finding container 51c45f7c2f60d3298f8a8081c170b8fc8b23987ef306e67c43764e8dd0dbd94e: Status 404 returned error can't find the container with id 51c45f7c2f60d3298f8a8081c170b8fc8b23987ef306e67c43764e8dd0dbd94e Oct 30 19:41:48.185128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:48.185095 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" Oct 30 19:41:48.212910 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:48.212858 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bjxr8" event={"ID":"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d","Type":"ContainerStarted","Data":"864df2188a691e271f31877f4b6be2a13283310e7558bdddb98dd61c5811edd5"} Oct 30 19:41:48.212910 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:48.212910 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bjxr8" event={"ID":"8cdfc4e4-12ba-463b-aabc-e66376c6fb8d","Type":"ContainerStarted","Data":"51c45f7c2f60d3298f8a8081c170b8fc8b23987ef306e67c43764e8dd0dbd94e"} Oct 30 19:41:48.260055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:48.260001 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bjxr8" podStartSLOduration=2.259981312 podStartE2EDuration="2.259981312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:48.258917452 +0000 UTC m=+375.708410177" watchObservedRunningTime="2026-10-30 19:41:48.259981312 +0000 UTC m=+375.709474048" Oct 30 19:41:50.867583 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:41:50.867545 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:41:51.227354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:51.227313 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" event={"ID":"b1bda3e4-bb2b-4269-9ddf-0c536750dfc1","Type":"ContainerStarted","Data":"8b91d5e8ab8df8381977462c5ff23d8b7ea6f2c3926fb1e75e5020395c6125d8"} Oct 30 19:41:51.266489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:51.266442 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mg72b" podStartSLOduration=2.332634049 podStartE2EDuration="7.266427871s" totalImagesPullingTime="4.933793822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:51.265210689 +0000 UTC m=+378.714703417" watchObservedRunningTime="2026-10-30 19:41:51.266427871 +0000 UTC m=+378.715920596" Oct 30 19:41:52.990959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:52.990928 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7"] Oct 30 19:41:52.995856 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:52.995835 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.017951 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.017920 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:41:53.023913 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.023888 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:41:53.031742 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.031718 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7"] Oct 30 19:41:53.065559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.065530 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:41:53.065746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.065727 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:41:53.065855 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.065820 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vcfxs\"" Oct 30 19:41:53.072265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.072235 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6839fdf-2c95-4b36-9919-eac9d385123a-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.072394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.072269 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6839fdf-2c95-4b36-9919-eac9d385123a-tmp\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.072441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.072404 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkzw\" (UniqueName: \"kubernetes.io/projected/f6839fdf-2c95-4b36-9919-eac9d385123a-kube-api-access-gkkzw\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.173113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.173079 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkzw\" (UniqueName: \"kubernetes.io/projected/f6839fdf-2c95-4b36-9919-eac9d385123a-kube-api-access-gkkzw\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.173274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.173133 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6839fdf-2c95-4b36-9919-eac9d385123a-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.173327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.173303 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6839fdf-2c95-4b36-9919-eac9d385123a-tmp\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.175787 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.175758 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6839fdf-2c95-4b36-9919-eac9d385123a-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.175929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.175758 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6839fdf-2c95-4b36-9919-eac9d385123a-tmp\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.196196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.196167 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkzw\" (UniqueName: \"kubernetes.io/projected/f6839fdf-2c95-4b36-9919-eac9d385123a-kube-api-access-gkkzw\") pod \"kube-auth-proxy-66b985bdbd-mlfq7\" (UID: \"f6839fdf-2c95-4b36-9919-eac9d385123a\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.308201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.308123 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" Oct 30 19:41:53.509311 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.509277 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:41:53.513970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.513946 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.533319 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.533296 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:41:53.536720 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.536701 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:41:53.576791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.576724 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdlk\" (UniqueName: \"kubernetes.io/projected/76bafc1c-67f0-4b74-b191-6484890bbc1d-kube-api-access-qrdlk\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.576791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.576773 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76bafc1c-67f0-4b74-b191-6484890bbc1d-config-file\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.609460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.609431 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7"] Oct 30 19:41:53.612675 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:53.612651 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6839fdf_2c95_4b36_9919_eac9d385123a.slice/crio-e4ac34ff0bd645d2aeff76c2ca3388afce6d3cec9aeafa9ac40c20b49cf98eeb WatchSource:0}: Error finding container e4ac34ff0bd645d2aeff76c2ca3388afce6d3cec9aeafa9ac40c20b49cf98eeb: Status 404 returned error can't find the container with id e4ac34ff0bd645d2aeff76c2ca3388afce6d3cec9aeafa9ac40c20b49cf98eeb Oct 30 19:41:53.615319 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.615296 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:41:53.678191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.678158 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdlk\" (UniqueName: \"kubernetes.io/projected/76bafc1c-67f0-4b74-b191-6484890bbc1d-kube-api-access-qrdlk\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.678191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.678197 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76bafc1c-67f0-4b74-b191-6484890bbc1d-config-file\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.678841 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.678822 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76bafc1c-67f0-4b74-b191-6484890bbc1d-config-file\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.695600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.695569 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdlk\" (UniqueName: \"kubernetes.io/projected/76bafc1c-67f0-4b74-b191-6484890bbc1d-kube-api-access-qrdlk\") pod \"limitador-limitador-7c9b87d574-4whww\" (UID: \"76bafc1c-67f0-4b74-b191-6484890bbc1d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:53.826720 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:53.826694 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:54.123425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:54.123257 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:41:54.127487 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:41:54.127455 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bafc1c_67f0_4b74_b191_6484890bbc1d.slice/crio-8457be125b5cc696c0562c16902903d64a58d500328e9ce31d8372522117dd69 WatchSource:0}: Error finding container 8457be125b5cc696c0562c16902903d64a58d500328e9ce31d8372522117dd69: Status 404 returned error can't find the container with id 8457be125b5cc696c0562c16902903d64a58d500328e9ce31d8372522117dd69 Oct 30 19:41:54.240811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:54.240764 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" event={"ID":"f6839fdf-2c95-4b36-9919-eac9d385123a","Type":"ContainerStarted","Data":"e4ac34ff0bd645d2aeff76c2ca3388afce6d3cec9aeafa9ac40c20b49cf98eeb"} Oct 30 19:41:54.242470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:54.242445 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" event={"ID":"76bafc1c-67f0-4b74-b191-6484890bbc1d","Type":"ContainerStarted","Data":"8457be125b5cc696c0562c16902903d64a58d500328e9ce31d8372522117dd69"} Oct 30 19:41:57.258243 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:57.258206 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" event={"ID":"f6839fdf-2c95-4b36-9919-eac9d385123a","Type":"ContainerStarted","Data":"25a11f22f0abdca7d1651a7474102d1b0b8e5098d1680b818b02c54a790cc27d"} Oct 30 19:41:57.259626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:57.259602 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" event={"ID":"76bafc1c-67f0-4b74-b191-6484890bbc1d","Type":"ContainerStarted","Data":"66dbb1184deba3259bc53fabd80614289f6119a1d7470cafc70c4da3fd08af47"} Oct 30 19:41:57.259747 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:57.259733 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:41:57.297618 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:57.297559 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-mlfq7" podStartSLOduration=2.368252065 podStartE2EDuration="5.297541005s" totalImagesPullingTime="2.92928894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:57.296393801 +0000 UTC m=+384.745886527" watchObservedRunningTime="2026-10-30 19:41:57.297541005 +0000 UTC m=+384.747033730" Oct 30 19:41:57.341662 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:41:57.341612 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" podStartSLOduration=1.876303369 podStartE2EDuration="4.341594704s" totalImagesPullingTime="2.465291335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:57.341164674 +0000 UTC m=+384.790657401" watchObservedRunningTime="2026-10-30 19:41:57.341594704 +0000 UTC m=+384.791087426" Oct 30 19:42:01.120356 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.120321 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9"] Oct 30 19:42:01.123992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.123976 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.138321 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.138295 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-v49pw\"" Oct 30 19:42:01.148537 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.148518 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:42:01.148638 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.148608 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:42:01.149999 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.149978 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/2860740e-daff-4305-afaf-262de7a22bee-kube-api-access-tf9lz\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.150079 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.150012 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.150221 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.150200 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.150303 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.150245 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2860740e-daff-4305-afaf-262de7a22bee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.169647 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.169616 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:42:01.170695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.170680 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9"] Oct 30 19:42:01.251177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251134 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.251387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251184 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2860740e-daff-4305-afaf-262de7a22bee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.251387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251220 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/2860740e-daff-4305-afaf-262de7a22bee-kube-api-access-tf9lz\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.251387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251253 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.251613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251591 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.251674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.251645 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2860740e-daff-4305-afaf-262de7a22bee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.252276 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.252256 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2860740e-daff-4305-afaf-262de7a22bee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.268821 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.268791 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/2860740e-daff-4305-afaf-262de7a22bee-kube-api-access-tf9lz\") pod \"kserve-module-controller-manager-7566cf5fc8-lxcg9\" (UID: \"2860740e-daff-4305-afaf-262de7a22bee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.437039 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.436951 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:01.752406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:01.752350 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9"] Oct 30 19:42:01.755814 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:42:01.755788 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2860740e_daff_4305_afaf_262de7a22bee.slice/crio-9fc273e04c21b06e5533611318418f32b4f8ad30d8229ab264cd56fa87c825ac WatchSource:0}: Error finding container 9fc273e04c21b06e5533611318418f32b4f8ad30d8229ab264cd56fa87c825ac: Status 404 returned error can't find the container with id 9fc273e04c21b06e5533611318418f32b4f8ad30d8229ab264cd56fa87c825ac Oct 30 19:42:02.282829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:02.282795 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" event={"ID":"2860740e-daff-4305-afaf-262de7a22bee","Type":"ContainerStarted","Data":"9fc273e04c21b06e5533611318418f32b4f8ad30d8229ab264cd56fa87c825ac"} Oct 30 19:42:04.292792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:04.292754 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" event={"ID":"2860740e-daff-4305-afaf-262de7a22bee","Type":"ContainerStarted","Data":"72b41d3ce8e82adc7a5de72b1f81721d12b3e1964b5a6fed6a972d8461df2e2e"} Oct 30 19:42:04.293213 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:04.292853 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:04.342331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:04.342278 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" podStartSLOduration=1.8082516910000002 podStartE2EDuration="3.342261738s" totalImagesPullingTime="1.534010047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:04.342086959 +0000 UTC m=+391.791579684" watchObservedRunningTime="2026-10-30 19:42:04.342261738 +0000 UTC m=+391.791754464" Oct 30 19:42:08.264892 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:08.264860 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4whww" Oct 30 19:42:20.868333 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:42:20.868294 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:42:25.298414 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:25.298355 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lxcg9" Oct 30 19:42:50.867238 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:42:50.867192 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:42:58.900929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.900892 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rlcr5"] Oct 30 19:42:58.908263 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.908237 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:58.931698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.931677 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 19:42:58.932296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.931752 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l6whr\"" Oct 30 19:42:58.933634 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.933609 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rlcr5"] Oct 30 19:42:58.969858 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.969828 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfxp\" (UniqueName: \"kubernetes.io/projected/589ad651-f4dd-484b-92c0-5179357485bf-kube-api-access-skfxp\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:58.970028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:58.969873 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589ad651-f4dd-484b-92c0-5179357485bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.071409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.071375 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfxp\" (UniqueName: \"kubernetes.io/projected/589ad651-f4dd-484b-92c0-5179357485bf-kube-api-access-skfxp\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.071604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.071433 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589ad651-f4dd-484b-92c0-5179357485bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.073900 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.073876 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589ad651-f4dd-484b-92c0-5179357485bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.096055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.096024 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfxp\" (UniqueName: \"kubernetes.io/projected/589ad651-f4dd-484b-92c0-5179357485bf-kube-api-access-skfxp\") pod \"model-serving-api-75bd89fdfc-rlcr5\" (UID: \"589ad651-f4dd-484b-92c0-5179357485bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.221411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.221374 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:42:59.518548 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.518467 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rlcr5"] Oct 30 19:42:59.521949 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:42:59.521921 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589ad651_f4dd_484b_92c0_5179357485bf.slice/crio-dbe7772c004f68319a68ab5281b54fd1530ff9980163af0a6b2784a4c144a61a WatchSource:0}: Error finding container dbe7772c004f68319a68ab5281b54fd1530ff9980163af0a6b2784a4c144a61a: Status 404 returned error can't find the container with id dbe7772c004f68319a68ab5281b54fd1530ff9980163af0a6b2784a4c144a61a Oct 30 19:42:59.543199 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:42:59.543170 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" event={"ID":"589ad651-f4dd-484b-92c0-5179357485bf","Type":"ContainerStarted","Data":"dbe7772c004f68319a68ab5281b54fd1530ff9980163af0a6b2784a4c144a61a"} Oct 30 19:43:01.554780 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:01.554744 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" event={"ID":"589ad651-f4dd-484b-92c0-5179357485bf","Type":"ContainerStarted","Data":"d052f7e1229a732818f424250369c7d370449759207c1032e6069cddb34e1040"} Oct 30 19:43:01.555172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:01.554840 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:43:01.601435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:01.599901 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" podStartSLOduration=1.9044563970000001 podStartE2EDuration="3.599883901s" totalImagesPullingTime="1.695427504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:01.596805303 +0000 UTC m=+449.046298025" watchObservedRunningTime="2026-10-30 19:43:01.599883901 +0000 UTC m=+449.049376628" Oct 30 19:43:12.563819 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:12.563785 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-rlcr5" Oct 30 19:43:12.646654 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:12.646622 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:43:20.867067 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:43:20.866991 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:43:21.092624 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.092591 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 19:43:21.095406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.095387 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.118916 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.118855 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:43:21.119059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.119046 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcrrd\"" Oct 30 19:43:21.119100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.119058 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:43:21.122997 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.122975 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 19:43:21.171990 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.171963 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:43:21.171990 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.171963 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:43:21.180409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.180355 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nnq\" (UniqueName: \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.180491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.180420 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.180529 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.180481 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.180571 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.180539 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.281075 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.281040 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nnq\" (UniqueName: \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.281075 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.281079 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.281289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.281205 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.281327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.281316 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.281920 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.281897 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.283488 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.283466 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.283600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.283582 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.302229 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.302199 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nnq\" (UniqueName: \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\") pod \"maas-api-f7476c9bf-slr29\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.407704 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.407625 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:21.708319 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:21.708235 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 19:43:21.712310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:43:21.712284 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a7b7ab_3445_4878_ba81_277a85be52c4.slice/crio-7555d535ffe3febda60b21ba98c311c56e7b4f19c89827bbbd4c06b8988a334f WatchSource:0}: Error finding container 7555d535ffe3febda60b21ba98c311c56e7b4f19c89827bbbd4c06b8988a334f: Status 404 returned error can't find the container with id 7555d535ffe3febda60b21ba98c311c56e7b4f19c89827bbbd4c06b8988a334f Oct 30 19:43:22.651896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:22.651850 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" event={"ID":"a1a7b7ab-3445-4878-ba81-277a85be52c4","Type":"ContainerStarted","Data":"7555d535ffe3febda60b21ba98c311c56e7b4f19c89827bbbd4c06b8988a334f"} Oct 30 19:43:24.662187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:24.662151 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" event={"ID":"a1a7b7ab-3445-4878-ba81-277a85be52c4","Type":"ContainerStarted","Data":"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd"} Oct 30 19:43:24.662640 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:24.662275 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:24.702634 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:24.702581 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" podStartSLOduration=1.528646965 podStartE2EDuration="3.702566794s" totalImagesPullingTime="2.173919829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:24.701638035 +0000 UTC m=+472.151130767" watchObservedRunningTime="2026-10-30 19:43:24.702566794 +0000 UTC m=+472.152059553" Oct 30 19:43:30.676541 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:43:30.676516 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 19:43:50.866731 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:43:50.866693 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:44:20.868258 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:44:20.868218 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:44:50.866798 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:44:50.866755 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:44:55.571187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.571155 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh"] Oct 30 19:44:55.575081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.575061 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.599845 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.599818 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-674xl\"" Oct 30 19:44:55.599970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.599939 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:44:55.600042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.599973 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:44:55.600042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.600012 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:44:55.601656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.601636 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh"] Oct 30 19:44:55.656809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.656776 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.656809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.656815 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.657045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.656915 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.657045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.657012 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.657045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.657037 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vh7\" (UniqueName: \"kubernetes.io/projected/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-kube-api-access-l5vh7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.657210 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.657152 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758662 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758618 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758662 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758662 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vh7\" (UniqueName: \"kubernetes.io/projected/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-kube-api-access-l5vh7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758704 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758753 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758769 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.758918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.758803 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.759193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.759168 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.759274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.759205 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.759274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.759225 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.761145 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.761121 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.761453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.761429 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.784092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.784057 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vh7\" (UniqueName: \"kubernetes.io/projected/0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36-kube-api-access-l5vh7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh\" (UID: \"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:55.887116 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:55.887027 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:56.181141 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:56.181048 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh"] Oct 30 19:44:56.185157 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:44:56.185129 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ce8e3d2_4c44_4d20_8f68_0bd9b5a83c36.slice/crio-748573bd17e42f8fc408dfd99da949d3a17dfec6bd07f46421b6bd56df72f93d WatchSource:0}: Error finding container 748573bd17e42f8fc408dfd99da949d3a17dfec6bd07f46421b6bd56df72f93d: Status 404 returned error can't find the container with id 748573bd17e42f8fc408dfd99da949d3a17dfec6bd07f46421b6bd56df72f93d Oct 30 19:44:56.792033 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:56.791996 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:44:57.077662 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.077557 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" event={"ID":"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36","Type":"ContainerStarted","Data":"748573bd17e42f8fc408dfd99da949d3a17dfec6bd07f46421b6bd56df72f93d"} Oct 30 19:44:57.621228 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.620743 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw"] Oct 30 19:44:57.625220 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.625195 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.639177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.639152 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 19:44:57.656670 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.656638 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw"] Oct 30 19:44:57.676029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.675995 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.676253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.676063 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/ab0d2e72-c6c8-482c-b69f-9fd413e01569-kube-api-access-fbwwp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.676253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.676173 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.676399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.676288 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.676399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.676332 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.676399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.676358 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.776955 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.776914 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.776978 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/ab0d2e72-c6c8-482c-b69f-9fd413e01569-kube-api-access-fbwwp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777020 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777062 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777111 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777134 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777573 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777675 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777635 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.777717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.777678 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.779480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.779459 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0d2e72-c6c8-482c-b69f-9fd413e01569-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.779659 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.779642 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0d2e72-c6c8-482c-b69f-9fd413e01569-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.806838 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.806801 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/ab0d2e72-c6c8-482c-b69f-9fd413e01569-kube-api-access-fbwwp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw\" (UID: \"ab0d2e72-c6c8-482c-b69f-9fd413e01569\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:57.940701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:57.940611 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:44:58.083905 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:58.083866 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" event={"ID":"0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36","Type":"ContainerStarted","Data":"261ca6623322d2d21d0a54d06279868a50b0de26290bbac9feba9df0c2680606"} Oct 30 19:44:58.124995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:58.124942 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podStartSLOduration=2.099912156 podStartE2EDuration="3.124924269s" totalImagesPullingTime="1.025012113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:58.124121016 +0000 UTC m=+565.573613741" watchObservedRunningTime="2026-10-30 19:44:58.124924269 +0000 UTC m=+565.574416995" Oct 30 19:44:58.241238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:58.241207 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw"] Oct 30 19:44:58.244540 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:44:58.244512 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0d2e72_c6c8_482c_b69f_9fd413e01569.slice/crio-cb3df8f5c134eccdc644776167f6e03a2e20ab8d4ece6914ab746b5856420900 WatchSource:0}: Error finding container cb3df8f5c134eccdc644776167f6e03a2e20ab8d4ece6914ab746b5856420900: Status 404 returned error can't find the container with id cb3df8f5c134eccdc644776167f6e03a2e20ab8d4ece6914ab746b5856420900 Oct 30 19:44:58.887473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:58.887344 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:44:59.089503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:59.089462 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" event={"ID":"ab0d2e72-c6c8-482c-b69f-9fd413e01569","Type":"ContainerStarted","Data":"e17e0baaf2eb8b91a07ae1175c73074108a8a127af462c73c8e4703dd883f868"} Oct 30 19:44:59.089683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:59.089516 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" event={"ID":"ab0d2e72-c6c8-482c-b69f-9fd413e01569","Type":"ContainerStarted","Data":"cb3df8f5c134eccdc644776167f6e03a2e20ab8d4ece6914ab746b5856420900"} Oct 30 19:44:59.145939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:59.145833 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" podStartSLOduration=1.958142814 podStartE2EDuration="2.145818867s" totalImagesPullingTime="187.676053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:59.143887783 +0000 UTC m=+566.593380520" watchObservedRunningTime="2026-10-30 19:44:59.145818867 +0000 UTC m=+566.595311591" Oct 30 19:44:59.941459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:44:59.941412 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:45:00.751068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:00.751026 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:45:01.094147 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.094044 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf"] Oct 30 19:45:01.098156 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.098129 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.112133 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.112110 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:45:01.139714 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.139680 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf"] Oct 30 19:45:01.211037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211000 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.211037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211033 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a539a2-c29a-47c4-93bd-ebc74ad85856-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.211289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211055 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.211289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211200 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.211289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211235 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.211445 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.211376 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphsk\" (UniqueName: \"kubernetes.io/projected/47a539a2-c29a-47c4-93bd-ebc74ad85856-kube-api-access-kphsk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312323 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312275 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphsk\" (UniqueName: \"kubernetes.io/projected/47a539a2-c29a-47c4-93bd-ebc74ad85856-kube-api-access-kphsk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312341 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312390 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a539a2-c29a-47c4-93bd-ebc74ad85856-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312417 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312457 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312559 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312488 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312803 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312725 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312803 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.312865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.312832 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.314732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.314712 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a539a2-c29a-47c4-93bd-ebc74ad85856-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.315019 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.315001 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a539a2-c29a-47c4-93bd-ebc74ad85856-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.333247 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.333216 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphsk\" (UniqueName: \"kubernetes.io/projected/47a539a2-c29a-47c4-93bd-ebc74ad85856-kube-api-access-kphsk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bf7cf\" (UID: \"47a539a2-c29a-47c4-93bd-ebc74ad85856\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.411541 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.411454 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:01.717059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:01.716981 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf"] Oct 30 19:45:01.720825 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:01.720790 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a539a2_c29a_47c4_93bd_ebc74ad85856.slice/crio-a9db2524713148acd141de9835b67901dfc74f16c8f07ab80bb4e0890cacc9d2 WatchSource:0}: Error finding container a9db2524713148acd141de9835b67901dfc74f16c8f07ab80bb4e0890cacc9d2: Status 404 returned error can't find the container with id a9db2524713148acd141de9835b67901dfc74f16c8f07ab80bb4e0890cacc9d2 Oct 30 19:45:02.105589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.105554 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" event={"ID":"47a539a2-c29a-47c4-93bd-ebc74ad85856","Type":"ContainerStarted","Data":"9514b93de8bb6c0141f041e5250b313f0368a57c4be0800b06cfb85548caa4e2"} Oct 30 19:45:02.105992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.105615 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" event={"ID":"47a539a2-c29a-47c4-93bd-ebc74ad85856","Type":"ContainerStarted","Data":"a9db2524713148acd141de9835b67901dfc74f16c8f07ab80bb4e0890cacc9d2"} Oct 30 19:45:02.146713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.146660 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" podStartSLOduration=0.958276838 podStartE2EDuration="1.146641888s" totalImagesPullingTime="188.36505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.145873399 +0000 UTC m=+569.595366124" watchObservedRunningTime="2026-10-30 19:45:02.146641888 +0000 UTC m=+569.596134641" Oct 30 19:45:02.412443 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.412353 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:02.551767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.551729 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls"] Oct 30 19:45:02.555874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.555851 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.572713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.572685 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:45:02.580886 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.580853 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls"] Oct 30 19:45:02.624806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.624773 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789b903b-be64-4bdd-bf2c-20dd94fbce79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.624992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.624831 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.624992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.624923 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf6f\" (UniqueName: \"kubernetes.io/projected/789b903b-be64-4bdd-bf2c-20dd94fbce79-kube-api-access-mtf6f\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.624992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.624962 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.625111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.625064 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.625152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.625112 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.725894 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.725861 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789b903b-be64-4bdd-bf2c-20dd94fbce79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726151 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726124 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726429 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf6f\" (UniqueName: \"kubernetes.io/projected/789b903b-be64-4bdd-bf2c-20dd94fbce79-kube-api-access-mtf6f\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726499 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726625 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726584 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726728 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.726943 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.726914 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.727223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.727199 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.727739 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.727713 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.728669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.728645 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789b903b-be64-4bdd-bf2c-20dd94fbce79-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.728921 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.728905 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789b903b-be64-4bdd-bf2c-20dd94fbce79-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.745646 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.745610 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf6f\" (UniqueName: \"kubernetes.io/projected/789b903b-be64-4bdd-bf2c-20dd94fbce79-kube-api-access-mtf6f\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-td6ls\" (UID: \"789b903b-be64-4bdd-bf2c-20dd94fbce79\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:02.870990 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:02.870943 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:03.171866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:03.171782 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls"] Oct 30 19:45:03.174980 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:03.174951 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789b903b_be64_4bdd_bf2c_20dd94fbce79.slice/crio-8e94916ea924a7df0decb865020d0badfe7243a1e2dacde1278821796a8b4ef5 WatchSource:0}: Error finding container 8e94916ea924a7df0decb865020d0badfe7243a1e2dacde1278821796a8b4ef5: Status 404 returned error can't find the container with id 8e94916ea924a7df0decb865020d0badfe7243a1e2dacde1278821796a8b4ef5 Oct 30 19:45:04.116925 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:04.116884 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" event={"ID":"789b903b-be64-4bdd-bf2c-20dd94fbce79","Type":"ContainerStarted","Data":"519e38c397d216684de5d0ddc7bb859128e5bde97fd3c9cf601fb8c03d8b921e"} Oct 30 19:45:04.117114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:04.116952 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" event={"ID":"789b903b-be64-4bdd-bf2c-20dd94fbce79","Type":"ContainerStarted","Data":"8e94916ea924a7df0decb865020d0badfe7243a1e2dacde1278821796a8b4ef5"} Oct 30 19:45:04.871112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:04.871069 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:05.888042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:05.887996 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:45:05.900927 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:05.900898 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:45:05.941251 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:05.941194 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" podStartSLOduration=3.733499121 podStartE2EDuration="3.941178974s" totalImagesPullingTime="207.679853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.170683129 +0000 UTC m=+571.620175853" watchObservedRunningTime="2026-10-30 19:45:05.941178974 +0000 UTC m=+573.390671699" Oct 30 19:45:06.019080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.019049 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ngh9x"] Oct 30 19:45:06.023068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.023040 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.037067 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.037043 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 19:45:06.044739 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.044707 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ngh9x"] Oct 30 19:45:06.062291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062256 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-home\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.062500 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062425 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph46l\" (UniqueName: \"kubernetes.io/projected/45d0120c-b448-490e-ae2b-040abdd254c3-kube-api-access-ph46l\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.062500 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062467 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.062620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062603 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.062692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062630 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.062692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.062652 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d0120c-b448-490e-ae2b-040abdd254c3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.137177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.137141 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:45:06.164272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164181 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph46l\" (UniqueName: \"kubernetes.io/projected/45d0120c-b448-490e-ae2b-040abdd254c3-kube-api-access-ph46l\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164242 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164325 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164348 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164392 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d0120c-b448-490e-ae2b-040abdd254c3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164436 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-home\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164806 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.164833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164836 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-home\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.165013 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.164806 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.166637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.166616 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d0120c-b448-490e-ae2b-040abdd254c3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.166831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.166813 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d0120c-b448-490e-ae2b-040abdd254c3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.189442 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.189414 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph46l\" (UniqueName: \"kubernetes.io/projected/45d0120c-b448-490e-ae2b-040abdd254c3-kube-api-access-ph46l\") pod \"echo-simulator-kserve-64b6f8f96c-ngh9x\" (UID: \"45d0120c-b448-490e-ae2b-040abdd254c3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.339972 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.339935 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:06.639184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:06.639148 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ngh9x"] Oct 30 19:45:06.642220 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:06.642189 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d0120c_b448_490e_ae2b_040abdd254c3.slice/crio-a2d104c2e3c9c8d21a94676f9a368b4eef90ffc9f9faa51ee237df0220d4f2ca WatchSource:0}: Error finding container a2d104c2e3c9c8d21a94676f9a368b4eef90ffc9f9faa51ee237df0220d4f2ca: Status 404 returned error can't find the container with id a2d104c2e3c9c8d21a94676f9a368b4eef90ffc9f9faa51ee237df0220d4f2ca Oct 30 19:45:07.133978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.133939 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" event={"ID":"45d0120c-b448-490e-ae2b-040abdd254c3","Type":"ContainerStarted","Data":"a2d104c2e3c9c8d21a94676f9a368b4eef90ffc9f9faa51ee237df0220d4f2ca"} Oct 30 19:45:07.368497 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.368463 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:45:07.685646 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.683904 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq"] Oct 30 19:45:07.696978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.696949 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.711715 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.711688 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:45:07.724447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.724418 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq"] Oct 30 19:45:07.781115 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781073 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4pj\" (UniqueName: \"kubernetes.io/projected/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-kube-api-access-dn4pj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.781286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781173 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.781286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781243 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.781424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781334 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.781424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781412 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.781500 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.781444 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.882708 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882673 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.882880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882728 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.882880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882771 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.882880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882844 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4pj\" (UniqueName: \"kubernetes.io/projected/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-kube-api-access-dn4pj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.883054 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882885 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.883054 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.882928 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.883143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.883109 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.883326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.883302 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.883465 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.883423 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.885231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.885203 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.885530 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.885511 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.907094 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.907062 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4pj\" (UniqueName: \"kubernetes.io/projected/62b90662-f37f-4ba0-a9fc-9b37bcdfe838-kube-api-access-dn4pj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq\" (UID: \"62b90662-f37f-4ba0-a9fc-9b37bcdfe838\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:07.941482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.941406 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:45:07.956466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:07.956431 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:45:08.012256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:08.012218 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:08.163578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:08.163546 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" Oct 30 19:45:08.326983 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:08.326955 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq"] Oct 30 19:45:08.330571 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:08.330542 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b90662_f37f_4ba0_a9fc_9b37bcdfe838.slice/crio-6605521e933a9e550e02d42dd5275af2693d9e21bdfd78014a1a68b81e100537 WatchSource:0}: Error finding container 6605521e933a9e550e02d42dd5275af2693d9e21bdfd78014a1a68b81e100537: Status 404 returned error can't find the container with id 6605521e933a9e550e02d42dd5275af2693d9e21bdfd78014a1a68b81e100537 Oct 30 19:45:09.146908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.146869 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" event={"ID":"45d0120c-b448-490e-ae2b-040abdd254c3","Type":"ContainerStarted","Data":"6f99176675a9968e583ffb6a3e19a02e1bd221c1dc5008c6934752c182bd5524"} Oct 30 19:45:09.148539 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.148514 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" event={"ID":"62b90662-f37f-4ba0-a9fc-9b37bcdfe838","Type":"ContainerStarted","Data":"c6a7f1b032fc5dba4b1d74b28df5c4d0507ee0446466902e7dea156710f90b1b"} Oct 30 19:45:09.148681 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.148555 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" event={"ID":"62b90662-f37f-4ba0-a9fc-9b37bcdfe838","Type":"ContainerStarted","Data":"6605521e933a9e550e02d42dd5275af2693d9e21bdfd78014a1a68b81e100537"} Oct 30 19:45:09.206271 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.206214 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" podStartSLOduration=2.55924679 podStartE2EDuration="4.206201306s" totalImagesPullingTime="1.646954516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:09.20506336 +0000 UTC m=+576.654556085" watchObservedRunningTime="2026-10-30 19:45:09.206201306 +0000 UTC m=+576.655694031" Oct 30 19:45:09.276565 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.276497 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" podStartSLOduration=2.072678349 podStartE2EDuration="2.276476485s" totalImagesPullingTime="203.798136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:09.274582694 +0000 UTC m=+576.724075419" watchObservedRunningTime="2026-10-30 19:45:09.276476485 +0000 UTC m=+576.725969213" Oct 30 19:45:09.340750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:09.340698 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:10.013160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:10.013122 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:11.411666 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:11.411634 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:11.424142 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:11.424114 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:12.174703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:12.174673 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" Oct 30 19:45:12.871770 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:12.871729 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:12.884487 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:12.884451 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:13.180324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:13.180252 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" Oct 30 19:45:14.334644 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:14.334607 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:45:16.340330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:16.340290 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:16.353002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:16.352975 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:17.200133 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:17.200104 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:45:18.012479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:18.012437 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:18.025059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:18.025027 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:18.203983 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:18.203953 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" Oct 30 19:45:20.867971 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:45:20.867928 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:45:22.553789 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.553757 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:45:22.557923 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.557898 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.586158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.586126 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:45:22.598214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.598183 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:45:22.636051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.636020 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-94wtv\"" Oct 30 19:45:22.735420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735357 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735463 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735502 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5t8r\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735660 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735695 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735781 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735817 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735923 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735875 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.735957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.735919 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837266 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837316 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837518 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837584 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837617 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837778 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837762 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837826 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5t8r\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837832 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837906 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.837959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.837873 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.838226 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.838086 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.838226 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.838127 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.838451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.838286 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.838520 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.838479 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.840549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.840527 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.840742 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.840723 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.866455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.866427 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:22.872471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:22.872445 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5t8r\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wscc4\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:23.171561 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:23.171470 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:23.496177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:23.496152 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:45:23.498510 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:23.498480 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51753c8d_594b_480e_8a26_6fc135c47696.slice/crio-81e184b73562559fc9ebebbc34588431fe73aabaac844d4e4c7850471fbeb65c WatchSource:0}: Error finding container 81e184b73562559fc9ebebbc34588431fe73aabaac844d4e4c7850471fbeb65c: Status 404 returned error can't find the container with id 81e184b73562559fc9ebebbc34588431fe73aabaac844d4e4c7850471fbeb65c Oct 30 19:45:24.226029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:24.225990 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" event={"ID":"51753c8d-594b-480e-8a26-6fc135c47696","Type":"ContainerStarted","Data":"81e184b73562559fc9ebebbc34588431fe73aabaac844d4e4c7850471fbeb65c"} Oct 30 19:45:25.979701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:25.979672 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:25.979977 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:25.979739 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:25.979977 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:25.979768 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:26.292405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:26.291678 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" event={"ID":"51753c8d-594b-480e-8a26-6fc135c47696","Type":"ContainerStarted","Data":"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73"} Oct 30 19:45:26.359185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:26.359118 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podStartSLOduration=1.880938668 podStartE2EDuration="4.359100968s" totalImagesPullingTime="2.4781623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.357877709 +0000 UTC m=+593.807370435" watchObservedRunningTime="2026-10-30 19:45:26.359100968 +0000 UTC m=+593.808593694" Oct 30 19:45:27.181388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:27.174434 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:27.186385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:27.183017 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:27.186385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:27.183093 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:28.189627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:28.189592 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:28.190216 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:28.190182 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:29.131635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:29.131598 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:45:29.173658 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:29.173624 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:29.173904 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:29.173879 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:30.189534 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:30.189492 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:30.190057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:30.189555 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:31.189617 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:31.189576 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:31.190159 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:31.189640 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:32.172486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:32.172452 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:32.172653 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:32.172512 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:32.745540 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:32.745507 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:45:32.745997 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:32.745926 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:45:33.172182 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:33.172093 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:45:33.172182 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:33.172162 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 19:45:34.601740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.601703 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6vn57"] Oct 30 19:45:34.608620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.608596 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.628788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.628762 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6vn57"] Oct 30 19:45:34.642960 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.642936 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q2ntt\"" Oct 30 19:45:34.642960 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.642938 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:45:34.669620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.669583 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3a5c43-a952-42f9-9e67-3a629aca6b72-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.669763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.669645 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rxz\" (UniqueName: \"kubernetes.io/projected/3d3a5c43-a952-42f9-9e67-3a629aca6b72-kube-api-access-g8rxz\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.770487 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.770456 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3a5c43-a952-42f9-9e67-3a629aca6b72-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.770693 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.770523 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rxz\" (UniqueName: \"kubernetes.io/projected/3d3a5c43-a952-42f9-9e67-3a629aca6b72-kube-api-access-g8rxz\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.771430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.771406 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3a5c43-a952-42f9-9e67-3a629aca6b72-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.789873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.789840 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rxz\" (UniqueName: \"kubernetes.io/projected/3d3a5c43-a952-42f9-9e67-3a629aca6b72-kube-api-access-g8rxz\") pod \"payload-processing-5f78687d5b-6vn57\" (UID: \"3d3a5c43-a952-42f9-9e67-3a629aca6b72\") " pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:34.922263 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:34.922172 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:35.172632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:35.172545 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:35.172632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:35.172603 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:35.235642 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:35.235605 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6vn57"] Oct 30 19:45:35.240157 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:35.240126 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3a5c43_a952_42f9_9e67_3a629aca6b72.slice/crio-c14e331cf84f4639850b135b84793807e88c91c555897f6f906da5ecb18cf020 WatchSource:0}: Error finding container c14e331cf84f4639850b135b84793807e88c91c555897f6f906da5ecb18cf020: Status 404 returned error can't find the container with id c14e331cf84f4639850b135b84793807e88c91c555897f6f906da5ecb18cf020 Oct 30 19:45:35.344405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:35.338021 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" event={"ID":"3d3a5c43-a952-42f9-9e67-3a629aca6b72","Type":"ContainerStarted","Data":"c14e331cf84f4639850b135b84793807e88c91c555897f6f906da5ecb18cf020"} Oct 30 19:45:36.218578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:36.218535 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wscc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.219008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:36.218620 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.530476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:36.530441 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:36.531026 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:36.530921 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:36.531822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:36.531799 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:45:37.573064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.573030 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:45:37.576953 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.576927 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.590755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.590732 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 19:45:37.595045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.595021 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.603256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603217 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.603463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603281 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.603463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603320 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.603463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603379 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.603463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603426 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xhx\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.604091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603470 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.604091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603496 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.604091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603565 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.604091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603615 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.604091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.603832 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:45:37.626244 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.626199 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 19:45:37.704833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.704753 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bcf\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.704833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.704792 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.704833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.704817 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705107 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.704935 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705107 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705028 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705107 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705089 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705128 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705174 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705220 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xhx\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705265 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705293 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705356 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705425 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705455 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705500 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705523 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705539 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705590 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705600 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.705752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705679 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.706029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.705929 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.706114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.706092 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.707322 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.707302 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.707761 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.707744 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.728944 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.728916 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.729152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.729132 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xhx\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\") pod \"maas-default-gateway-openshift-default-75bf867b6c-jt5xb\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.806478 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806436 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806478 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806480 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806503 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806529 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806612 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806651 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806702 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bcf\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806735 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.806984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.806780 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.807079 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.807037 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.807408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.807375 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.807408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.807392 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.807593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.807496 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.807593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.807508 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.808810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.808789 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.809231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.809213 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.824251 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.824220 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.824339 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.824253 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bcf\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9xsqp\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:37.891438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.891402 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:37.932316 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:37.932272 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:38.200158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.200121 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:45:38.204332 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:38.204307 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ece28d_bb64_4681_8ce1_a68619cec5da.slice/crio-ad7eaa07c5d2da1146fe8bb6db46e8f2492f4d0837e0858245d721ef19586399 WatchSource:0}: Error finding container ad7eaa07c5d2da1146fe8bb6db46e8f2492f4d0837e0858245d721ef19586399: Status 404 returned error can't find the container with id ad7eaa07c5d2da1146fe8bb6db46e8f2492f4d0837e0858245d721ef19586399 Oct 30 19:45:38.207045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.207018 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.207115 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.207076 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.207115 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.207102 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.239712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.239668 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 19:45:38.242582 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:45:38.242556 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e969921_f44b_49eb_b81b_fb3686f1d130.slice/crio-5a1cd732c4337a12014c31bd199afada54e79a5c62c96f5de6a594730f15a7b2 WatchSource:0}: Error finding container 5a1cd732c4337a12014c31bd199afada54e79a5c62c96f5de6a594730f15a7b2: Status 404 returned error can't find the container with id 5a1cd732c4337a12014c31bd199afada54e79a5c62c96f5de6a594730f15a7b2 Oct 30 19:45:38.244922 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.244891 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.245005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.244958 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.245005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.244987 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:38.357104 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.357060 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" event={"ID":"1e969921-f44b-49eb-b81b-fb3686f1d130","Type":"ContainerStarted","Data":"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e"} Oct 30 19:45:38.357286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.357128 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" event={"ID":"1e969921-f44b-49eb-b81b-fb3686f1d130","Type":"ContainerStarted","Data":"5a1cd732c4337a12014c31bd199afada54e79a5c62c96f5de6a594730f15a7b2"} Oct 30 19:45:38.360017 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.359891 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" event={"ID":"d8ece28d-bb64-4681-8ce1-a68619cec5da","Type":"ContainerStarted","Data":"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c"} Oct 30 19:45:38.360017 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.359948 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" event={"ID":"d8ece28d-bb64-4681-8ce1-a68619cec5da","Type":"ContainerStarted","Data":"ad7eaa07c5d2da1146fe8bb6db46e8f2492f4d0837e0858245d721ef19586399"} Oct 30 19:45:38.362119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.362007 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" event={"ID":"3d3a5c43-a952-42f9-9e67-3a629aca6b72","Type":"ContainerStarted","Data":"b7cf76cfd719bc23f508f654f478668ffde58092bca4b33597da3bd3cf8cc5b5"} Oct 30 19:45:38.362119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.362054 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:38.405547 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.405446 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podStartSLOduration=1.405426348 podStartE2EDuration="1.405426348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:38.404184395 +0000 UTC m=+605.853677133" watchObservedRunningTime="2026-10-30 19:45:38.405426348 +0000 UTC m=+605.854919069" Oct 30 19:45:38.449136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.448762 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podStartSLOduration=1.448742134 podStartE2EDuration="1.448742134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:38.447537416 +0000 UTC m=+605.897030154" watchObservedRunningTime="2026-10-30 19:45:38.448742134 +0000 UTC m=+605.898234860" Oct 30 19:45:38.490964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.490852 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" podStartSLOduration=2.292843869 podStartE2EDuration="4.49083682s" totalImagesPullingTime="2.197992951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:38.490186116 +0000 UTC m=+605.939678841" watchObservedRunningTime="2026-10-30 19:45:38.49083682 +0000 UTC m=+605.940329545" Oct 30 19:45:38.891632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.891536 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:38.932958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:38.932924 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:39.896392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:39.895418 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:39.896392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:39.895493 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:39.934132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:39.934097 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:39.934332 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:39.934155 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:40.893389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:40.892798 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:40.893389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:40.892877 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:40.932942 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:40.932902 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:40.933427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:40.932974 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:41.892857 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:41.892811 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:41.893193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:41.893162 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:41.933703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:41.933663 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:41.934061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:41.933716 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:42.892845 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:42.892798 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:42.893155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:42.893122 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:42.933052 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:42.933012 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:42.933285 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:42.933263 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:43.894346 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:43.893834 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:43.894346 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:43.893919 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:43.933804 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:43.933763 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:43.933973 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:43.933839 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:44.892471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:44.892429 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-jt5xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:45:44.892626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:44.892516 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:45:44.933425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:44.933389 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:44.933825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:44.933453 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:45.933125 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:45.933081 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:45:45.933308 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:45.933151 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:45:46.680334 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.680305 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:46.680848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.680712 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:46.681442 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.681420 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:45:46.936907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.936834 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:46.937148 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.937129 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:46.937927 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:46.937908 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 19:45:49.379703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:45:49.379671 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" Oct 30 19:45:50.867904 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:45:50.867866 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:46:20.868162 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:46:20.868034 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:46:26.656268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.656223 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 19:46:26.660529 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.660502 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.677967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.677937 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5db436-openshift-default-dockercfg-n7492\"" Oct 30 19:46:26.690936 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.690902 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 19:46:26.859120 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859062 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859127 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859217 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859263 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859308 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpbp\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859511 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859330 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859511 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859356 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859574 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859525 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.859712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.859684 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960426 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960317 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpbp\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960426 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960356 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960426 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960419 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960441 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960471 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960508 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960531 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960594 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.960709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960653 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.961002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960840 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.961002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960923 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.961002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.960941 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.961096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.961004 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.961275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.961256 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.963017 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.962996 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.963089 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.963000 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.996430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.996390 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpbp\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:26.997118 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:26.997100 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:27.118598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.118559 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 19:46:27.123128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.123108 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.144790 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.144760 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a3426-openshift-default-dockercfg-4xlnw\"" Oct 30 19:46:27.155333 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.155302 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 19:46:27.162419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162350 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162527 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162566 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162637 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162665 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162685 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162804 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162854 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8nq\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.162979 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.162891 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.263805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263767 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.263805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263819 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263845 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263868 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263925 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263964 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8nq\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.263987 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264032 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264140 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264316 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264400 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264737 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.264855 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264839 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.265003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.264985 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.266806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.266780 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.267017 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.267000 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.278723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.278697 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:27.282192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.281874 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.282192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.281959 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8nq\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.437568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.437530 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:27.613645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.613468 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 19:46:27.616643 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:27.616614 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7336305a_07e9_4976_8ea0_65248a79a04d.slice/crio-c384fd3ce46dea809eacb9eb0f05a6108f072219e8cf9ec650c41f438d51c5f8 WatchSource:0}: Error finding container c384fd3ce46dea809eacb9eb0f05a6108f072219e8cf9ec650c41f438d51c5f8: Status 404 returned error can't find the container with id c384fd3ce46dea809eacb9eb0f05a6108f072219e8cf9ec650c41f438d51c5f8 Oct 30 19:46:27.618964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.618931 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:27.619055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.619011 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:27.619055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.619046 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:27.651648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.651613 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerStarted","Data":"c384fd3ce46dea809eacb9eb0f05a6108f072219e8cf9ec650c41f438d51c5f8"} Oct 30 19:46:27.740532 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.740497 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 19:46:27.744389 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:27.744327 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25a782d_e324_4336_b93d_bc292e630f5e.slice/crio-03bb95ad3eec823bc8a5e67b2c7292cd5b33cbe319baa76c5250e808c0176628 WatchSource:0}: Error finding container 03bb95ad3eec823bc8a5e67b2c7292cd5b33cbe319baa76c5250e808c0176628: Status 404 returned error can't find the container with id 03bb95ad3eec823bc8a5e67b2c7292cd5b33cbe319baa76c5250e808c0176628 Oct 30 19:46:27.746956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.746921 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:27.747034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.746991 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:27.747034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:27.747024 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:28.658320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:28.658270 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerStarted","Data":"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602"} Oct 30 19:46:28.660330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:28.660303 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerStarted","Data":"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14"} Oct 30 19:46:28.660499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:28.660353 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerStarted","Data":"03bb95ad3eec823bc8a5e67b2c7292cd5b33cbe319baa76c5250e808c0176628"} Oct 30 19:46:28.729117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:28.729058 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podStartSLOduration=2.7290435779999997 podStartE2EDuration="2.729043578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:28.717760888 +0000 UTC m=+656.167253610" watchObservedRunningTime="2026-10-30 19:46:28.729043578 +0000 UTC m=+656.178536301" Oct 30 19:46:28.767096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:28.767044 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podStartSLOduration=1.767029188 podStartE2EDuration="1.767029188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:28.765349337 +0000 UTC m=+656.214842051" watchObservedRunningTime="2026-10-30 19:46:28.767029188 +0000 UTC m=+656.216521911" Oct 30 19:46:29.279516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.279413 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:29.285968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.285937 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:29.438651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.438618 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:29.443724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.443696 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:29.666032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.665944 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:29.666032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.666004 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:29.666687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.666667 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:46:29.667041 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:29.667025 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:46:31.885985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.885949 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 19:46:31.890612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.890591 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:31.914302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.914276 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 19:46:31.918062 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.918032 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn8d\" (UniqueName: \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:31.918198 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.918183 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:31.918249 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.918238 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:31.918302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.918272 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:31.924611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:31.924586 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-f349a1-serving-cert\"" Oct 30 19:46:32.019452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.019415 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn8d\" (UniqueName: \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.019643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.019526 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.019643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.019587 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.019643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.019620 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.020329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.020307 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.023386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.023337 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.027884 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.027861 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.041348 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.041323 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn8d\" (UniqueName: \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\") pod \"maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.204231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.204155 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:32.532104 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.532079 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 19:46:32.534822 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:32.534793 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode811c39f_15d9_4189_a386_b3c8db802672.slice/crio-4d019a92cba4aab2000eed874d0084d8e6d06389c2d4c2c10a32abcb9f79df7b WatchSource:0}: Error finding container 4d019a92cba4aab2000eed874d0084d8e6d06389c2d4c2c10a32abcb9f79df7b: Status 404 returned error can't find the container with id 4d019a92cba4aab2000eed874d0084d8e6d06389c2d4c2c10a32abcb9f79df7b Oct 30 19:46:32.536267 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.536248 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:46:32.681740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:32.681710 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" event={"ID":"e811c39f-15d9-4189-a386-b3c8db802672","Type":"ContainerStarted","Data":"4d019a92cba4aab2000eed874d0084d8e6d06389c2d4c2c10a32abcb9f79df7b"} Oct 30 19:46:33.344350 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.344260 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 19:46:33.348458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.348427 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.379695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.379667 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-284bf6\"" Oct 30 19:46:33.379831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.379696 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-284bf6-dockercfg-67ggm\"" Oct 30 19:46:33.382002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.381977 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 19:46:33.436226 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.436186 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm96c\" (UniqueName: \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.436389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.436239 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.537908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.537861 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm96c\" (UniqueName: \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.537908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.537910 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.538643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.538621 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.563992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.563959 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm96c\" (UniqueName: \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\") pod \"payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.663620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.663531 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:33.689777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.689738 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" event={"ID":"e811c39f-15d9-4189-a386-b3c8db802672","Type":"ContainerStarted","Data":"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a"} Oct 30 19:46:33.690043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.690023 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:33.741953 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.741905 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" podStartSLOduration=2.278707077 podStartE2EDuration="2.741890787s" totalImagesPullingTime="463.18371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:33.741260784 +0000 UTC m=+661.190753510" watchObservedRunningTime="2026-10-30 19:46:33.741890787 +0000 UTC m=+661.191383569" Oct 30 19:46:33.967709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:33.967624 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 19:46:33.973184 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:33.973146 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf006e551_4480_4475_a270_1d803ab8259e.slice/crio-4659796134e119cae8d1ad4e5242733b68c7d549a8c6fd1133fc9bea5baee767 WatchSource:0}: Error finding container 4659796134e119cae8d1ad4e5242733b68c7d549a8c6fd1133fc9bea5baee767: Status 404 returned error can't find the container with id 4659796134e119cae8d1ad4e5242733b68c7d549a8c6fd1133fc9bea5baee767 Oct 30 19:46:34.342598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.342561 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 19:46:34.346342 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.346319 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.377390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.373014 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6c4dd1\"" Oct 30 19:46:34.388746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.388712 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 19:46:34.448468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.448428 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ckr\" (UniqueName: \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.448468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.448467 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.457901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.457878 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6c4dd1-dockercfg-hvfnv\"" Oct 30 19:46:34.550258 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.550221 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ckr\" (UniqueName: \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.550450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.550264 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.551073 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.551046 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.573871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.573835 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ckr\" (UniqueName: \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\") pod \"payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.666203 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.666124 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:34.697339 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.697290 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" event={"ID":"f006e551-4480-4475-a270-1d803ab8259e","Type":"ContainerStarted","Data":"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae"} Oct 30 19:46:34.697549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.697388 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" event={"ID":"f006e551-4480-4475-a270-1d803ab8259e","Type":"ContainerStarted","Data":"4659796134e119cae8d1ad4e5242733b68c7d549a8c6fd1133fc9bea5baee767"} Oct 30 19:46:34.697629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.697611 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:34.736261 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.736204 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" podStartSLOduration=1.736188053 podStartE2EDuration="1.736188053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:34.735326518 +0000 UTC m=+662.184819242" watchObservedRunningTime="2026-10-30 19:46:34.736188053 +0000 UTC m=+662.185680777" Oct 30 19:46:34.967546 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:34.967519 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 19:46:34.970271 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:34.970241 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bae6318_f6d7_4874_a331_784d18707f48.slice/crio-0137578ebf449ab5b1197d4bed90200cc87a3eb0e908fb8a2b477200e96825c5 WatchSource:0}: Error finding container 0137578ebf449ab5b1197d4bed90200cc87a3eb0e908fb8a2b477200e96825c5: Status 404 returned error can't find the container with id 0137578ebf449ab5b1197d4bed90200cc87a3eb0e908fb8a2b477200e96825c5 Oct 30 19:46:35.405321 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.405280 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 19:46:35.409325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.409298 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.427146 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.427118 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-01f7d3\"" Oct 30 19:46:35.429727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.429695 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 19:46:35.437121 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.437092 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-01f7d3-dockercfg-gjr7s\"" Oct 30 19:46:35.461029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.460996 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.461205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.461187 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cn9\" (UniqueName: \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.496702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.496412 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 19:46:35.501809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.501777 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.518459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.518432 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 19:46:35.562523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.562490 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.562695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.562551 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.562695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.562675 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cn9\" (UniqueName: \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.562767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.562704 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949ks\" (UniqueName: \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.563203 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.563179 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.580907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.580884 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cn9\" (UniqueName: \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\") pod \"payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.664433 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.664319 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.664433 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.664423 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949ks\" (UniqueName: \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.665061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.665039 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.688048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.688016 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949ks\" (UniqueName: \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\") pod \"payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:35.704241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.704206 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" event={"ID":"7bae6318-f6d7-4874-a331-784d18707f48","Type":"ContainerStarted","Data":"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961"} Oct 30 19:46:35.704419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.704270 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" event={"ID":"7bae6318-f6d7-4874-a331-784d18707f48","Type":"ContainerStarted","Data":"0137578ebf449ab5b1197d4bed90200cc87a3eb0e908fb8a2b477200e96825c5"} Oct 30 19:46:35.704488 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.704465 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:35.723768 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.723734 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:35.759440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.758696 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" podStartSLOduration=1.758676978 podStartE2EDuration="1.758676978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.757792968 +0000 UTC m=+663.207285693" watchObservedRunningTime="2026-10-30 19:46:35.758676978 +0000 UTC m=+663.208169703" Oct 30 19:46:35.818585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:35.818556 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:36.047684 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.047653 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 19:46:36.052117 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:36.052083 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a40f54_dfd4_41b8_86e5_045f359c2be5.slice/crio-9f6e65c889647e2468b34ee273552bef70b373402b4dbf25bec3a7ec1a5b273b WatchSource:0}: Error finding container 9f6e65c889647e2468b34ee273552bef70b373402b4dbf25bec3a7ec1a5b273b: Status 404 returned error can't find the container with id 9f6e65c889647e2468b34ee273552bef70b373402b4dbf25bec3a7ec1a5b273b Oct 30 19:46:36.154091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.154051 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 19:46:36.166881 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:36.166848 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cc3275_277b_4536_8ae6_7db2e9126f37.slice/crio-34a8e092d5de4637ccd934315c2ab838d1ffaee02c7e651b4b9a47a03ed3d13f WatchSource:0}: Error finding container 34a8e092d5de4637ccd934315c2ab838d1ffaee02c7e651b4b9a47a03ed3d13f: Status 404 returned error can't find the container with id 34a8e092d5de4637ccd934315c2ab838d1ffaee02c7e651b4b9a47a03ed3d13f Oct 30 19:46:36.415732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.414252 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 19:46:36.418385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.418350 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.439981 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.438611 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 19:46:36.473008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.472971 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.473188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.473124 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrfn\" (UniqueName: \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.496536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.496501 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-109cfe\"" Oct 30 19:46:36.498726 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.498696 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-109cfe-dockercfg-698c2\"" Oct 30 19:46:36.574822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.574788 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrfn\" (UniqueName: \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.574995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.574882 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.575672 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.575653 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.592944 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.592920 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrfn\" (UniqueName: \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\") pod \"payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.711425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.711317 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" event={"ID":"23a40f54-dfd4-41b8-86e5-045f359c2be5","Type":"ContainerStarted","Data":"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7"} Oct 30 19:46:36.711425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.711407 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:36.711656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.711434 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" event={"ID":"23a40f54-dfd4-41b8-86e5-045f359c2be5","Type":"ContainerStarted","Data":"9f6e65c889647e2468b34ee273552bef70b373402b4dbf25bec3a7ec1a5b273b"} Oct 30 19:46:36.713008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.712982 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" event={"ID":"03cc3275-277b-4536-8ae6-7db2e9126f37","Type":"ContainerStarted","Data":"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e"} Oct 30 19:46:36.713123 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.713032 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" event={"ID":"03cc3275-277b-4536-8ae6-7db2e9126f37","Type":"ContainerStarted","Data":"34a8e092d5de4637ccd934315c2ab838d1ffaee02c7e651b4b9a47a03ed3d13f"} Oct 30 19:46:36.713173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.713120 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:36.736776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.736751 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:36.750315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.750258 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" podStartSLOduration=1.750240526 podStartE2EDuration="1.750240526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.747223363 +0000 UTC m=+664.196716088" watchObservedRunningTime="2026-10-30 19:46:36.750240526 +0000 UTC m=+664.199733261" Oct 30 19:46:36.782618 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:36.782552 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" podStartSLOduration=1.782534583 podStartE2EDuration="1.782534583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.782486786 +0000 UTC m=+664.231979512" watchObservedRunningTime="2026-10-30 19:46:36.782534583 +0000 UTC m=+664.232027308" Oct 30 19:46:37.035744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.035711 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 19:46:37.039014 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:37.038971 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e306930_f287_4d59_ad46_9ae4eaeb4f50.slice/crio-ed33723febaa8d40b9c17375b2433e512a24d4d4952f0e0c85cc2812de6d87a0 WatchSource:0}: Error finding container ed33723febaa8d40b9c17375b2433e512a24d4d4952f0e0c85cc2812de6d87a0: Status 404 returned error can't find the container with id ed33723febaa8d40b9c17375b2433e512a24d4d4952f0e0c85cc2812de6d87a0 Oct 30 19:46:37.213580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.213547 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:46:37.295134 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.295004 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 19:46:37.299219 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.299189 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.312115 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.312089 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5db436\"" Oct 30 19:46:37.317149 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.317127 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5db436-dockercfg-qxnmr\"" Oct 30 19:46:37.324619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.324588 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 19:46:37.383575 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.383537 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.383802 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.383594 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kghx\" (UniqueName: \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.485034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.484994 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.485465 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.485049 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kghx\" (UniqueName: \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.486082 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.486059 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.506644 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.506610 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kghx\" (UniqueName: \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\") pod \"payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.612887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.612783 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:37.721132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.721096 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" event={"ID":"2e306930-f287-4d59-ad46-9ae4eaeb4f50","Type":"ContainerStarted","Data":"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49"} Oct 30 19:46:37.721286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.721269 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:37.721394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.721347 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" event={"ID":"2e306930-f287-4d59-ad46-9ae4eaeb4f50","Type":"ContainerStarted","Data":"ed33723febaa8d40b9c17375b2433e512a24d4d4952f0e0c85cc2812de6d87a0"} Oct 30 19:46:37.758918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.758863 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" podStartSLOduration=1.7588471430000001 podStartE2EDuration="1.758847143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:37.757083589 +0000 UTC m=+665.206576314" watchObservedRunningTime="2026-10-30 19:46:37.758847143 +0000 UTC m=+665.208339867" Oct 30 19:46:37.914971 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:37.914527 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 19:46:37.917353 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:37.917326 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798aec19_6a8e_4bdf_81a6_d35e2a6717e6.slice/crio-162acc657c4ffc3542eddae1c60d290758377ff55b4d18514f55f9996335403c WatchSource:0}: Error finding container 162acc657c4ffc3542eddae1c60d290758377ff55b4d18514f55f9996335403c: Status 404 returned error can't find the container with id 162acc657c4ffc3542eddae1c60d290758377ff55b4d18514f55f9996335403c Oct 30 19:46:38.157111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.157078 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 19:46:38.160981 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.160960 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.174045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.174019 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8a3426\"" Oct 30 19:46:38.183174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.183100 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 19:46:38.193255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.193224 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.193550 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.193509 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvtf\" (UniqueName: \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.294743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.294710 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.294918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.294760 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvtf\" (UniqueName: \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.295448 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.295428 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.312404 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.312377 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvtf\" (UniqueName: \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\") pod \"payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.477290 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.477262 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:38.728287 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.728192 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" event={"ID":"798aec19-6a8e-4bdf-81a6-d35e2a6717e6","Type":"ContainerStarted","Data":"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357"} Oct 30 19:46:38.728287 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.728250 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" event={"ID":"798aec19-6a8e-4bdf-81a6-d35e2a6717e6","Type":"ContainerStarted","Data":"162acc657c4ffc3542eddae1c60d290758377ff55b4d18514f55f9996335403c"} Oct 30 19:46:38.728751 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.728535 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:38.768194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.768127 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" podStartSLOduration=1.7681101350000001 podStartE2EDuration="1.768110135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:38.764177853 +0000 UTC m=+666.213670575" watchObservedRunningTime="2026-10-30 19:46:38.768110135 +0000 UTC m=+666.217602861" Oct 30 19:46:38.775647 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:38.775623 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 19:46:38.776865 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:38.776839 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66499969_3c1f_4a18_816c_a30d6e470abb.slice/crio-4e187072df2808e07f51bb42a745b7d272fd7b94b7dae80d5afd4299ace81f42 WatchSource:0}: Error finding container 4e187072df2808e07f51bb42a745b7d272fd7b94b7dae80d5afd4299ace81f42: Status 404 returned error can't find the container with id 4e187072df2808e07f51bb42a745b7d272fd7b94b7dae80d5afd4299ace81f42 Oct 30 19:46:39.703621 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:39.703592 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 19:46:39.734418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:39.734384 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" event={"ID":"66499969-3c1f-4a18-816c-a30d6e470abb","Type":"ContainerStarted","Data":"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8"} Oct 30 19:46:39.734802 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:39.734435 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" event={"ID":"66499969-3c1f-4a18-816c-a30d6e470abb","Type":"ContainerStarted","Data":"4e187072df2808e07f51bb42a745b7d272fd7b94b7dae80d5afd4299ace81f42"} Oct 30 19:46:39.734802 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:39.734472 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:39.779169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:39.779116 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" podStartSLOduration=1.779102108 podStartE2EDuration="1.779102108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:39.776450268 +0000 UTC m=+667.225942993" watchObservedRunningTime="2026-10-30 19:46:39.779102108 +0000 UTC m=+667.228594831" Oct 30 19:46:41.464852 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.464820 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 19:46:41.469206 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.469180 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.489502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.489465 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-01f7d3-serving-cert\"" Oct 30 19:46:41.506528 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.506499 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 19:46:41.531427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.531402 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.531678 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.531450 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.531678 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.531494 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68bd\" (UniqueName: \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.531792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.531753 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.633154 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.633120 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.633154 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.633156 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68bd\" (UniqueName: \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.633341 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.633214 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.633436 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.633343 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.633892 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.633868 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.635588 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.635553 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.635752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.635731 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.651713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.651686 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68bd\" (UniqueName: \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\") pod \"maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:41.784951 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:41.784913 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:42.087032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:42.087002 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 19:46:42.088494 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:42.088471 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97b046d_1071_4817_8c94_403ecb2ed589.slice/crio-ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71 WatchSource:0}: Error finding container ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71: Status 404 returned error can't find the container with id ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71 Oct 30 19:46:42.756664 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:42.756619 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" event={"ID":"b97b046d-1071-4817-8c94-403ecb2ed589","Type":"ContainerStarted","Data":"5a939671ce154e141a0f29e58e4129d581b58a3dd332d8144885504b61064c52"} Oct 30 19:46:42.757143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:42.756682 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" event={"ID":"b97b046d-1071-4817-8c94-403ecb2ed589","Type":"ContainerStarted","Data":"ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71"} Oct 30 19:46:42.757143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:42.756731 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:42.801637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:42.801571 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" podStartSLOduration=1.504604938 podStartE2EDuration="1.801552836s" totalImagesPullingTime="296.947898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:42.799747927 +0000 UTC m=+670.249240651" watchObservedRunningTime="2026-10-30 19:46:42.801552836 +0000 UTC m=+670.251045560" Oct 30 19:46:45.707904 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:45.707868 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 19:46:46.715595 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:46.715564 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 19:46:47.724104 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:47.724078 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 19:46:47.724498 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:47.724142 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 19:46:48.077707 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.077624 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:46:48.732119 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.732091 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 19:46:48.754859 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.754825 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 19:46:48.759188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.759167 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:48.767510 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.767489 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 19:46:48.772129 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.772108 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5db436-serving-cert\"" Oct 30 19:46:48.782409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.782384 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 19:46:48.928056 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.928025 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:48.928222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.928176 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmnn\" (UniqueName: \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:48.928340 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.928324 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:48.928410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:48.928358 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.029706 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.029621 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.029706 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.029699 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmnn\" (UniqueName: \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.029934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.029785 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.029934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.029815 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.030454 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.030432 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.032082 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.032060 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.032169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.032151 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.047495 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.047466 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmnn\" (UniqueName: \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\") pod \"maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.073506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.073483 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.371960 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.371738 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 19:46:49.374878 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:49.374841 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0050383_e123_49fe_be8f_2583d1e0b036.slice/crio-c81d8a1f4a004df4e8e93f2bb699de6c3ab97336dd4823d35a08a63145695896 WatchSource:0}: Error finding container c81d8a1f4a004df4e8e93f2bb699de6c3ab97336dd4823d35a08a63145695896: Status 404 returned error can't find the container with id c81d8a1f4a004df4e8e93f2bb699de6c3ab97336dd4823d35a08a63145695896 Oct 30 19:46:49.738523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.738499 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 19:46:49.800194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.800154 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" event={"ID":"a0050383-e123-49fe-be8f-2583d1e0b036","Type":"ContainerStarted","Data":"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f"} Oct 30 19:46:49.800395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.800217 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" event={"ID":"a0050383-e123-49fe-be8f-2583d1e0b036","Type":"ContainerStarted","Data":"c81d8a1f4a004df4e8e93f2bb699de6c3ab97336dd4823d35a08a63145695896"} Oct 30 19:46:49.800395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.800377 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:49.839832 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:49.839774 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" podStartSLOduration=1.532737346 podStartE2EDuration="1.839759437s" totalImagesPullingTime="307.022091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:49.838602853 +0000 UTC m=+677.288095577" watchObservedRunningTime="2026-10-30 19:46:49.839759437 +0000 UTC m=+677.289252156" Oct 30 19:46:50.743576 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:50.743546 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 19:46:50.867755 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:46:50.867716 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:46:55.812420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:55.812355 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 19:46:56.574381 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.574326 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 19:46:56.578919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.578894 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.591911 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.591883 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-01f7d3-openshift-default-dockercfg-mt2rl\"" Oct 30 19:46:56.605534 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.605498 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 19:46:56.614635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614601 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614649 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rrh\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614682 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614708 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614739 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614769 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614888 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.614967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.614947 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.615037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.615022 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716134 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716103 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716164 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716227 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716274 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716302 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rrh\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716330 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716353 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716399 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716431 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716730 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716685 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.716793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.716769 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.717080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.717054 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.717191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.717140 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.717522 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.717500 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.718828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.718802 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.719273 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.719254 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.735506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.735479 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.735628 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.735576 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rrh\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:56.895430 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:56.895339 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:57.040323 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.040285 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:46:57.211004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.210809 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 19:46:57.212646 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:46:57.212621 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c55fa64_bfb5_44ff_a606_703ffb19007a.slice/crio-b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8 WatchSource:0}: Error finding container b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8: Status 404 returned error can't find the container with id b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8 Oct 30 19:46:57.215193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.215162 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:57.215272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.215227 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:57.215272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.215255 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:57.854481 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.854441 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" event={"ID":"3c55fa64-bfb5-44ff-a606-703ffb19007a","Type":"ContainerStarted","Data":"5492c66bb865df411a85676ee9f3ad717ff4d17d02a6d795d6352fc693bc598c"} Oct 30 19:46:57.854680 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.854506 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" event={"ID":"3c55fa64-bfb5-44ff-a606-703ffb19007a","Type":"ContainerStarted","Data":"b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8"} Oct 30 19:46:57.902806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.902774 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:46:57.942581 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:57.942529 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podStartSLOduration=1.942515032 podStartE2EDuration="1.942515032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:57.930739269 +0000 UTC m=+685.380231996" watchObservedRunningTime="2026-10-30 19:46:57.942515032 +0000 UTC m=+685.392007758" Oct 30 19:46:58.902651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:58.902609 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:46:58.902938 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:58.902909 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:46:59.901856 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:59.901810 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:46:59.903035 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:46:59.902994 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:47:00.937390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:00.912850 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:47:00.941393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:00.937988 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:47:01.907388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:01.906824 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:47:01.907388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:01.906892 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:47:02.931598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:02.923667 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:47:02.931598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:02.923736 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:47:03.901445 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:03.901414 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:47:03.901926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:03.901876 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:47:03.902786 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:03.902764 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 19:47:04.911095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:04.911046 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:47:07.084587 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.084555 5991 generic.go:356] "Generic (PLEG): container finished" podID="7336305a-07e9-4976-8ea0-65248a79a04d" containerID="0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602" exitCode=0 Oct 30 19:47:07.086413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.084641 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerDied","Data":"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602"} Oct 30 19:47:07.086413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.085060 5991 scope.go:122] "RemoveContainer" containerID="0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602" Oct 30 19:47:07.086776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.086452 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:07.086776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.086512 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:07.086776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:07.086545 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:08.100450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.100408 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerStarted","Data":"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46"} Oct 30 19:47:08.104542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.104487 5991 generic.go:356] "Generic (PLEG): container finished" podID="c25a782d-e324-4336-b93d-bc292e630f5e" containerID="2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14" exitCode=0 Oct 30 19:47:08.104699 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.104554 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerDied","Data":"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14"} Oct 30 19:47:08.105411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.105390 5991 scope.go:122] "RemoveContainer" containerID="2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14" Oct 30 19:47:08.106735 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.106707 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:08.106836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.106775 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:08.106836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.106808 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:08.293779 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.293751 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:47:08.294299 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.294277 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:47:08.294470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:08.294450 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:47:09.117392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.117325 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerStarted","Data":"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd"} Oct 30 19:47:09.285388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.284921 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:47:09.285388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.285019 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:47:09.437988 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.437902 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:47:09.438276 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.438207 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:47:09.438421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:09.438277 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:47:10.283799 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:10.283759 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:47:10.284491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:10.284453 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:47:10.438551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:10.438512 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:47:10.438743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:10.438574 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:47:11.300881 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:11.300840 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:47:11.305620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:11.305580 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:47:11.438534 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:11.438500 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:47:11.438709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:11.438558 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:47:12.284300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:12.283800 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:47:12.284300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:12.283871 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:47:12.437938 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:12.437903 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:47:12.438433 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:12.437959 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:47:13.419499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:13.418709 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:47:13.419499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:13.419402 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:47:13.420112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:13.420089 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 19:47:13.438712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:13.438674 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:47:13.439074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:13.438733 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:47:14.608423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:14.608385 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:47:14.609278 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:14.609251 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:47:14.610284 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:14.610265 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 19:47:18.363676 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:18.363640 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:47:20.872960 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:47:20.872925 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:47:24.259831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.259798 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 19:47:24.264167 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.264143 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.279915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.279892 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"simulator-w0-5db436-kserve-self-signed-certs\"" Oct 30 19:47:24.290438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.290398 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"openshift-service-ca.crt\"" Oct 30 19:47:24.290603 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.290438 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"kube-root-ca.crt\"" Oct 30 19:47:24.292650 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.292626 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 19:47:24.349505 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349465 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.349686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349533 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqm2k\" (UniqueName: \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.349686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349612 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.349686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349645 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.349814 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349797 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.349849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.349827 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.382185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.382157 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"default-dockercfg-jb4r7\"" Oct 30 19:47:24.450848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450812 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.450848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450852 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450897 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450930 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqm2k\" (UniqueName: \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450950 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.450966 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.451422 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.451562 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.451641 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.451619 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.453337 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.453312 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.453448 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.453345 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.470051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.470018 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqm2k\" (UniqueName: \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\") pod \"simulator-w0-5db436-kserve-6467b4f8bf-tqswl\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.577688 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.577526 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:24.877639 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:24.877610 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 19:47:24.879779 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:47:24.879744 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e7c0848_8746_483b_8ffb_f58640119bf8.slice/crio-f88e45d0724e5d2012e5b89ae2aa41e3f6388c7d86c6f29664dd1242114f46f3 WatchSource:0}: Error finding container f88e45d0724e5d2012e5b89ae2aa41e3f6388c7d86c6f29664dd1242114f46f3: Status 404 returned error can't find the container with id f88e45d0724e5d2012e5b89ae2aa41e3f6388c7d86c6f29664dd1242114f46f3 Oct 30 19:47:25.240931 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:25.240890 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" event={"ID":"4e7c0848-8746-483b-8ffb-f58640119bf8","Type":"ContainerStarted","Data":"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837"} Oct 30 19:47:25.241113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:25.240944 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" event={"ID":"4e7c0848-8746-483b-8ffb-f58640119bf8","Type":"ContainerStarted","Data":"f88e45d0724e5d2012e5b89ae2aa41e3f6388c7d86c6f29664dd1242114f46f3"} Oct 30 19:47:25.282215 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:25.282120 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podStartSLOduration=1.282100946 podStartE2EDuration="1.282100946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:25.280515906 +0000 UTC m=+712.730008634" watchObservedRunningTime="2026-10-30 19:47:25.282100946 +0000 UTC m=+712.731593672" Oct 30 19:47:25.578614 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:25.578531 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:28.876701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:28.876659 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:47:34.578717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:34.578677 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:34.591836 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:34.591807 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:35.309585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:35.309555 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:47:46.492178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:46.492090 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:47:50.869539 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:47:50.869497 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:47:56.990958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:56.990920 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 19:47:56.995383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:56.995338 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.014447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.014416 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6c4dd1-openshift-default-dockercfg-tw2l4\"" Oct 30 19:47:57.042411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.036380 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 19:47:57.143745 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.143705 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jn8l\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.143745 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.143752 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.143984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.143772 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.143984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.143906 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.143984 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.143935 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.144114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.144045 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.144169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.144144 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.144219 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.144185 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.144319 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.144302 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.245940 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jn8l\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.245986 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246006 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246040 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246066 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246118 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246246 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246291 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246675 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246423 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246675 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246609 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246784 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246698 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246784 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246735 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246881 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246798 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.246945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.246927 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.248584 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.248554 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.248961 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.248941 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.265058 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.265016 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.265239 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.265167 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jn8l\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.309704 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.309664 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:57.628228 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.628194 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 19:47:57.630234 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:47:57.630204 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf30b33_c1e5_45ca_aa90_643d7169571e.slice/crio-5c50b714149d12ede7798be78b7188b38e079e4dd216e52125ddbb42b976cbd4 WatchSource:0}: Error finding container 5c50b714149d12ede7798be78b7188b38e079e4dd216e52125ddbb42b976cbd4: Status 404 returned error can't find the container with id 5c50b714149d12ede7798be78b7188b38e079e4dd216e52125ddbb42b976cbd4 Oct 30 19:47:57.632603 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.632570 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:57.632691 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.632636 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:57.632691 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:57.632683 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:58.446140 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:58.446101 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" event={"ID":"1bf30b33-c1e5-45ca-aa90-643d7169571e","Type":"ContainerStarted","Data":"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757"} Oct 30 19:47:58.446683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:58.446160 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" event={"ID":"1bf30b33-c1e5-45ca-aa90-643d7169571e","Type":"ContainerStarted","Data":"5c50b714149d12ede7798be78b7188b38e079e4dd216e52125ddbb42b976cbd4"} Oct 30 19:47:59.310090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:59.310048 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:47:59.311957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:59.311932 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:47:59.312034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:47:59.311987 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:00.310719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:00.310677 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:48:00.311087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:00.310748 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:01.310179 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:01.310146 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:48:01.310455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:01.310433 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:02.310490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:02.310450 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:48:02.310894 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:02.310520 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:03.310538 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:03.310504 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:48:03.310940 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:03.310569 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:04.310792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:04.310752 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:48:04.311215 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:04.310813 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 19:48:05.315893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:05.315856 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:48:05.317152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:05.316858 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:48:05.318082 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:05.318057 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 19:48:05.385797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:05.385738 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podStartSLOduration=9.385721185 podStartE2EDuration="9.385721185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:58.526245136 +0000 UTC m=+745.975737861" watchObservedRunningTime="2026-10-30 19:48:05.385721185 +0000 UTC m=+752.835213910" Oct 30 19:48:06.879839 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.879795 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 19:48:06.885627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.885597 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.910235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.910197 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"kube-root-ca.crt\"" Oct 30 19:48:06.910235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.910214 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"openshift-service-ca.crt\"" Oct 30 19:48:06.910235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.910227 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"premium-w4-8a3426-kserve-self-signed-certs\"" Oct 30 19:48:06.913248 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.913219 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"default-dockercfg-ljjjh\"" Oct 30 19:48:06.915719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.915687 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 19:48:06.994939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.994892 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.995152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.994944 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.995152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.995041 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.995262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.995174 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.995262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.995230 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:06.995262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:06.995258 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097162 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097229 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097264 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097477 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097517 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097549 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.097985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097963 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.098057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.097971 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.098233 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.098214 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.099818 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.099788 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.100096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.100076 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.115157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.115119 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\") pod \"premium-w4-8a3426-kserve-64d764f4bb-wsb7n\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.201005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.200913 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:07.502160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.502130 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 19:48:07.504390 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:48:07.504340 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb42b2133_af7b_4713_966b_e98f5e14e3f4.slice/crio-e66f2318ad737f1657e2cda268fac8a268a680b704266570915eb746ee3537c6 WatchSource:0}: Error finding container e66f2318ad737f1657e2cda268fac8a268a680b704266570915eb746ee3537c6: Status 404 returned error can't find the container with id e66f2318ad737f1657e2cda268fac8a268a680b704266570915eb746ee3537c6 Oct 30 19:48:07.522794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:07.522762 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" event={"ID":"b42b2133-af7b-4713-966b-e98f5e14e3f4","Type":"ContainerStarted","Data":"e66f2318ad737f1657e2cda268fac8a268a680b704266570915eb746ee3537c6"} Oct 30 19:48:08.530302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:08.530268 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" event={"ID":"b42b2133-af7b-4713-966b-e98f5e14e3f4","Type":"ContainerStarted","Data":"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc"} Oct 30 19:48:08.569138 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:08.569074 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podStartSLOduration=2.569055929 podStartE2EDuration="2.569055929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:08.567707303 +0000 UTC m=+756.017200029" watchObservedRunningTime="2026-10-30 19:48:08.569055929 +0000 UTC m=+756.018548657" Oct 30 19:48:09.201672 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:09.201635 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:11.328325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.328290 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 19:48:11.332762 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.332734 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.357194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.357160 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"premium-w0-5db436-kserve-self-signed-certs\"" Oct 30 19:48:11.359796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.359759 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 19:48:11.451440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451399 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.451646 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451466 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ffc\" (UniqueName: \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.451646 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451587 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.451746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451662 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.451824 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451800 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.451862 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.451839 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552546 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552507 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552590 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552608 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552647 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552678 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ffc\" (UniqueName: \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.552927 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.552752 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.553033 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.553007 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.553083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.553069 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.553183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.553166 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.555012 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.554982 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.555160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.555111 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.570759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.570719 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ffc\" (UniqueName: \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\") pod \"premium-w0-5db436-kserve-7dcbb8ddb8-59gms\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.647133 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.647024 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:11.946888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:11.946854 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 19:48:11.948923 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:48:11.948894 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb22ca8_145d_4116_85bf_0ac9f44379e6.slice/crio-891247321d0e1620670d98545c50a49c69e77e0617853d23104cdcc369347880 WatchSource:0}: Error finding container 891247321d0e1620670d98545c50a49c69e77e0617853d23104cdcc369347880: Status 404 returned error can't find the container with id 891247321d0e1620670d98545c50a49c69e77e0617853d23104cdcc369347880 Oct 30 19:48:12.556335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:12.556298 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" event={"ID":"deb22ca8-145d-4116-85bf-0ac9f44379e6","Type":"ContainerStarted","Data":"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb"} Oct 30 19:48:12.556771 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:12.556345 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" event={"ID":"deb22ca8-145d-4116-85bf-0ac9f44379e6","Type":"ContainerStarted","Data":"891247321d0e1620670d98545c50a49c69e77e0617853d23104cdcc369347880"} Oct 30 19:48:12.598728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:12.598666 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podStartSLOduration=1.59864827 podStartE2EDuration="1.59864827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:12.596221411 +0000 UTC m=+760.045714135" watchObservedRunningTime="2026-10-30 19:48:12.59864827 +0000 UTC m=+760.048141037" Oct 30 19:48:12.647753 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:12.647713 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:17.201140 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:17.201099 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:17.214420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:17.214389 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:17.600413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:17.600348 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:48:18.535092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:18.535055 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:48:20.868662 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:48:20.868628 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:48:21.647772 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:21.647735 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:21.660509 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:21.660478 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:22.632008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:22.631978 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:48:50.868525 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:48:50.868489 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:48:51.608593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:51.608557 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:48:56.722770 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.722739 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 19:48:56.727215 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.727196 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.739833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.739814 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f349a1-openshift-default-dockercfg-n7zxx\"" Oct 30 19:48:56.752959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.752928 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 19:48:56.788097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788059 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788108 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788216 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwbd\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788277 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788655 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788695 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788855 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788952 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788939 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.788994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.788979 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.889901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.889859 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.889901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.889917 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.889937 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.889996 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890048 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890073 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890101 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890134 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwbd\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890415 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890572 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890708 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890686 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890998 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890970 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.890998 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.890988 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.891281 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.891259 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.892692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.892671 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.893625 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.893603 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.908420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.908388 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:56.908536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:56.908401 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwbd\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:57.040893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.040797 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:57.355235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.355210 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 19:48:57.358033 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:48:57.358005 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbd6c23_a3ad_43d3_a004_65e19d01ba60.slice/crio-9f6c361fa2e1b6e8a48b36d5f585c720c20ef64b71439e7699ad75fe64d02655 WatchSource:0}: Error finding container 9f6c361fa2e1b6e8a48b36d5f585c720c20ef64b71439e7699ad75fe64d02655: Status 404 returned error can't find the container with id 9f6c361fa2e1b6e8a48b36d5f585c720c20ef64b71439e7699ad75fe64d02655 Oct 30 19:48:57.360317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.360288 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:57.360407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.360353 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:57.360453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.360409 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:57.840479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.840436 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" event={"ID":"6bbd6c23-a3ad-43d3-a004-65e19d01ba60","Type":"ContainerStarted","Data":"b8a53cb70f4d184416e3fe1494df7eb4ad9a2f4cd083b7233fe98903b9833256"} Oct 30 19:48:57.840887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:57.840495 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" event={"ID":"6bbd6c23-a3ad-43d3-a004-65e19d01ba60","Type":"ContainerStarted","Data":"9f6c361fa2e1b6e8a48b36d5f585c720c20ef64b71439e7699ad75fe64d02655"} Oct 30 19:48:58.041873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:58.041834 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:48:58.042967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:58.042942 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:48:58.043097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:58.042988 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:48:59.042091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:59.042053 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:48:59.042501 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:48:59.042121 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:49:00.041603 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:00.041567 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:49:00.041784 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:00.041627 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:49:01.041350 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:01.041312 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:49:01.041829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:01.041395 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:49:02.041184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:02.041135 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:49:02.041184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:02.041189 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:49:03.041166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:03.041131 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:49:03.041347 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:03.041183 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:49:04.836381 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:04.836320 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:49:04.837129 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:04.837103 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:49:04.838232 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:04.838214 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 19:49:04.889253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:04.889187 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podStartSLOduration=8.889169708 podStartE2EDuration="8.889169708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:57.903774103 +0000 UTC m=+805.353266852" watchObservedRunningTime="2026-10-30 19:49:04.889169708 +0000 UTC m=+812.338662434" Oct 30 19:49:20.869540 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:49:20.869453 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:49:25.718171 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:25.718131 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:49:26.687206 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.687155 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 19:49:26.692670 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.692638 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.705933 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.705905 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-284bf6-openshift-default-dockercfg-fq2xw\"" Oct 30 19:49:26.739764 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.739732 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 19:49:26.829389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.806129 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.854389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.829723 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.854714 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.854681 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.863396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.859480 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.959676 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.959573 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74qk\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.961619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.961587 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.961931 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.961904 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.962212 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.962191 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:26.962401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:26.962383 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.087612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.087543 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.087945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.087922 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088162 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088353 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088336 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088483 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088635 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088789 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74qk\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.088960 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.088944 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.089111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.089092 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.089785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.089740 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.091023 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.090993 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.092073 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.091496 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.092229 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.091739 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.092338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.092009 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.113292 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.112897 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.122176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.122054 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74qk\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.160385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.154446 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.161858 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.161786 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.343269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.334506 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:27.453636 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.453586 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 19:49:27.461035 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.460011 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.496830 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.496885 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.496998 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497036 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497063 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqmd\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497145 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497193 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497228 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.497506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.497261 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.511036 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.510927 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 19:49:27.511697 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.511655 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-109cfe-openshift-default-dockercfg-4jtdp\"" Oct 30 19:49:27.598074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.597961 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598019 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598048 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598106 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598136 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598242 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598277 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598301 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqmd\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598413 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598554 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598563 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.598866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.598797 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.599328 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.599037 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.599328 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.599142 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.601293 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.601268 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.602761 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.602740 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.616411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.616346 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.616560 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.616373 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqmd\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:27.719916 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.719886 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 19:49:27.723119 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:49:27.723081 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee65e032_5d15_4462_ac7f_300575e6fc4d.slice/crio-d729c033bb8327dd611bae0aee1cf30baf54cc8d64c60d7441be0673322d8869 WatchSource:0}: Error finding container d729c033bb8327dd611bae0aee1cf30baf54cc8d64c60d7441be0673322d8869: Status 404 returned error can't find the container with id d729c033bb8327dd611bae0aee1cf30baf54cc8d64c60d7441be0673322d8869 Oct 30 19:49:27.726342 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.726301 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:27.726458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.726418 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:27.726504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.726472 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:27.792988 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:27.792954 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:28.072909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.072871 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" event={"ID":"ee65e032-5d15-4462-ac7f-300575e6fc4d","Type":"ContainerStarted","Data":"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997"} Oct 30 19:49:28.073096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.072957 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" event={"ID":"ee65e032-5d15-4462-ac7f-300575e6fc4d","Type":"ContainerStarted","Data":"d729c033bb8327dd611bae0aee1cf30baf54cc8d64c60d7441be0673322d8869"} Oct 30 19:49:28.122406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.120329 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podStartSLOduration=2.120312568 podStartE2EDuration="2.120312568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:28.117906188 +0000 UTC m=+835.567398926" watchObservedRunningTime="2026-10-30 19:49:28.120312568 +0000 UTC m=+835.569805295" Oct 30 19:49:28.142312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.142268 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 19:49:28.143632 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:49:28.143594 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63b3b88_539f_4c10_859a_904431939bb7.slice/crio-91bedb38fb1c988a77a2befee897222979f4d3f398244dd1ef7b2f46e00adfde WatchSource:0}: Error finding container 91bedb38fb1c988a77a2befee897222979f4d3f398244dd1ef7b2f46e00adfde: Status 404 returned error can't find the container with id 91bedb38fb1c988a77a2befee897222979f4d3f398244dd1ef7b2f46e00adfde Oct 30 19:49:28.146719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.146672 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:28.146865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.146750 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:28.146865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.146780 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:28.335037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.335008 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:28.336548 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.336526 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:28.336669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:28.336574 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:29.110635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.110520 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" event={"ID":"a63b3b88-539f-4c10-859a-904431939bb7","Type":"ContainerStarted","Data":"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332"} Oct 30 19:49:29.110635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.110592 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" event={"ID":"a63b3b88-539f-4c10-859a-904431939bb7","Type":"ContainerStarted","Data":"91bedb38fb1c988a77a2befee897222979f4d3f398244dd1ef7b2f46e00adfde"} Oct 30 19:49:29.156192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.156139 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podStartSLOduration=2.15612461 podStartE2EDuration="2.15612461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:29.151736391 +0000 UTC m=+836.601229118" watchObservedRunningTime="2026-10-30 19:49:29.15612461 +0000 UTC m=+836.605617340" Oct 30 19:49:29.336001 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.335962 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:29.336188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.336020 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:29.812386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.812247 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:29.812609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.812489 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:29.812609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:29.812569 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:30.335166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:30.335136 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:30.335593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:30.335199 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:30.795386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:30.794815 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:30.795386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:30.794882 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:31.335496 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:31.335459 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:31.335906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:31.335512 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:31.814389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:31.814149 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:31.814389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:31.814209 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:32.335255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:32.335207 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:32.335480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:32.335264 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:32.814390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:32.813768 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:32.814390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:32.813834 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:33.335431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:33.335395 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:33.335656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:33.335456 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:33.813577 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:33.813544 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:33.813816 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:33.813794 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:34.335038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:34.334995 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:34.335509 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:34.335054 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:49:34.794526 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:34.794490 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:49:34.794795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:34.794544 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:49:35.339174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.339142 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:35.339689 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.339566 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:35.340265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.340244 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 19:49:35.798744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.798714 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:35.799106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.799080 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:35.800026 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:35.800002 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 19:49:40.478262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.478216 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 19:49:40.483661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.483633 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.519440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.519409 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"distinct-w4-8a3426-kserve-self-signed-certs\"" Oct 30 19:49:40.522740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.522710 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 19:49:40.641439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641406 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.641651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641457 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.641722 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641655 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.641722 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641701 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gncj\" (UniqueName: \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.641808 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641775 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.641869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.641856 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743517 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743426 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743685 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743565 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743685 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743596 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gncj\" (UniqueName: \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743685 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743628 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743711 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.743851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.743842 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.744268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.744054 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.744268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.744108 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.744268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.744227 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.746545 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.746521 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.746658 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.746562 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.763508 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.763477 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gncj\" (UniqueName: \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\") pod \"distinct-w4-8a3426-kserve-8957c8df7-mf9fp\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:40.796345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:40.796312 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:41.123034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:41.123004 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 19:49:41.125274 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:49:41.125248 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e12231_4859_49d4_9c38_d485230c0a6b.slice/crio-a2bcda5cf41b86be8e776c7f5811707f9e28ecfbbc0c2216193f2389b19d0b4b WatchSource:0}: Error finding container a2bcda5cf41b86be8e776c7f5811707f9e28ecfbbc0c2216193f2389b19d0b4b: Status 404 returned error can't find the container with id a2bcda5cf41b86be8e776c7f5811707f9e28ecfbbc0c2216193f2389b19d0b4b Oct 30 19:49:41.209943 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:41.209915 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" event={"ID":"42e12231-4859-49d4-9c38-d485230c0a6b","Type":"ContainerStarted","Data":"a2bcda5cf41b86be8e776c7f5811707f9e28ecfbbc0c2216193f2389b19d0b4b"} Oct 30 19:49:42.050967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.050923 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 19:49:42.055147 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.055128 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058354 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058517 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058471 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058533 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058593 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058633 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ssf\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058750 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058819 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058802 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058850 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.058931 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.058890 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.120386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.119414 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160502 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160555 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160607 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ssf\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160756 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160838 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160867 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160927 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.160984 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.161013 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.162076 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.162151 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.161859 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.162315 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.162719 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.162396 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.164679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.164606 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.182278 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.179951 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.221490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.220969 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.233897 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.233803 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ssf\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.238418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.238345 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" event={"ID":"42e12231-4859-49d4-9c38-d485230c0a6b","Type":"ContainerStarted","Data":"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1"} Oct 30 19:49:42.291394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.290385 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 19:49:42.313339 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.307685 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.331388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.314253 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podStartSLOduration=2.314233861 podStartE2EDuration="2.314233861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:42.302297334 +0000 UTC m=+849.751790060" watchObservedRunningTime="2026-10-30 19:49:42.314233861 +0000 UTC m=+849.763726590" Oct 30 19:49:42.340702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.340389 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368078 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368312 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368378 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368443 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzq9\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368496 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368540 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368595 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368638 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.368671 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.375021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.374589 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.472936 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzq9\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.472989 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473016 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473051 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473077 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473102 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473153 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473234 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473265 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473656 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.473825 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.474002 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.474094 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.474795 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.474739 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.476782 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.476755 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.476888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.476872 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.491037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.490972 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.491671 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.491640 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzq9\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.647180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.647098 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:42.720622 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.720588 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 19:49:42.725256 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:49:42.725227 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cbf843_85b1_4096_af22_0a05e23a96c6.slice/crio-966edfb0698fb867c218444a35b41527fdf6f1f01f057c66756bd4065c2ddf53 WatchSource:0}: Error finding container 966edfb0698fb867c218444a35b41527fdf6f1f01f057c66756bd4065c2ddf53: Status 404 returned error can't find the container with id 966edfb0698fb867c218444a35b41527fdf6f1f01f057c66756bd4065c2ddf53 Oct 30 19:49:42.728587 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.728543 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:42.728859 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.728833 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:42.728999 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.728980 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:42.797390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.797331 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:42.953275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.953236 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 19:49:42.954539 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:49:42.954512 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27c4352_5fd8_45e6_ad8f_2771530b9100.slice/crio-6a6e41afb84a55799edfd066df8af84b65e95a1c0968c0115953518c40c5244f WatchSource:0}: Error finding container 6a6e41afb84a55799edfd066df8af84b65e95a1c0968c0115953518c40c5244f: Status 404 returned error can't find the container with id 6a6e41afb84a55799edfd066df8af84b65e95a1c0968c0115953518c40c5244f Oct 30 19:49:42.957291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.957259 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:42.957410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.957327 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:42.957410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:42.957355 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:43.247171 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.247135 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" event={"ID":"22cbf843-85b1-4096-af22-0a05e23a96c6","Type":"ContainerStarted","Data":"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092"} Oct 30 19:49:43.247672 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.247204 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" event={"ID":"22cbf843-85b1-4096-af22-0a05e23a96c6","Type":"ContainerStarted","Data":"966edfb0698fb867c218444a35b41527fdf6f1f01f057c66756bd4065c2ddf53"} Oct 30 19:49:43.249027 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.248998 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" event={"ID":"e27c4352-5fd8-45e6-ad8f-2771530b9100","Type":"ContainerStarted","Data":"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3"} Oct 30 19:49:43.249159 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.249038 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" event={"ID":"e27c4352-5fd8-45e6-ad8f-2771530b9100","Type":"ContainerStarted","Data":"6a6e41afb84a55799edfd066df8af84b65e95a1c0968c0115953518c40c5244f"} Oct 30 19:49:43.288895 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.288832 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podStartSLOduration=1.288812921 podStartE2EDuration="1.288812921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.286255359 +0000 UTC m=+850.735748110" watchObservedRunningTime="2026-10-30 19:49:43.288812921 +0000 UTC m=+850.738305649" Oct 30 19:49:43.326543 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.326486 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podStartSLOduration=1.326471847 podStartE2EDuration="1.326471847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.32640707 +0000 UTC m=+850.775899820" watchObservedRunningTime="2026-10-30 19:49:43.326471847 +0000 UTC m=+850.775964611" Oct 30 19:49:43.375850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.375815 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:43.377337 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.377305 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:43.377472 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.377411 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:43.647512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:43.647434 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:44.386390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:44.385836 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:44.386390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:44.385904 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:44.649072 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:44.648988 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:44.649236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:44.649071 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:45.383390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:45.377785 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:45.383390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:45.377853 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:45.648428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:45.648331 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:45.648428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:45.648411 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:46.379388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:46.378855 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:46.379388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:46.378926 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:46.647542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:46.647459 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:46.647542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:46.647523 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:47.385389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:47.381972 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:47.385389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:47.382056 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:47.648504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:47.648421 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:47.648504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:47.648484 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:48.375830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:48.375791 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:48.376094 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:48.376064 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:48.647851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:48.647757 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:48.647851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:48.647828 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:49.375426 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:49.375383 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:49.375616 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:49.375463 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:49:49.647637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:49.647542 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:49:49.647637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:49.647612 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:49:50.380470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.380438 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:50.380950 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.380883 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:50.381609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.381591 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 19:49:50.653004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.652923 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:50.653358 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.653337 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:50.654008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.653989 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 19:49:50.797441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.797404 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:50.810660 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:50.810632 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:50.873169 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:49:50.873121 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:49:51.363262 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:51.363228 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:49:58.130557 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:49:58.130517 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:50:12.616542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.616503 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 19:50:12.622674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.622648 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.651103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.651063 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 19:50:12.748006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.747964 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748032 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748050 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748072 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748264 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748285 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748306 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fc4x\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748476 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748443 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.748523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.748484 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850222 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850281 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850399 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850457 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850483 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850506 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850646 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850772 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850800 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850847 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.850968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.850850 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fc4x\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.851089 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.851067 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.851210 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.851189 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.851265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.851245 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.852998 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.852975 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.853257 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.853233 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.883254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.883157 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:50:12.884867 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.884837 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fc4x\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.885025 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.884867 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:12.889127 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.889108 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:12.917544 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.917512 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:50:12.937962 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:12.937885 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:13.054281 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054242 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054288 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054426 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054554 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054587 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjxb\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054678 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054715 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054738 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.054916 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.054816 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155667 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155732 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155817 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155844 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjxb\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155889 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155919 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155934 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.155993 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.155968 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156283 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156083 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156283 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156118 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156415 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156624 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156603 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156814 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156799 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.156915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.156894 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.158587 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.158568 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.158858 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.158837 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.175031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.174998 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjxb\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.175764 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.175741 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.221918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.221882 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:13.260310 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:13.260278 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf3d919_899c_4fcb_bbb8_704b70166626.slice/crio-589b56f20362d3a31c2794396430824d3df1036b97e2b27a03bc0611143bfd95 WatchSource:0}: Error finding container 589b56f20362d3a31c2794396430824d3df1036b97e2b27a03bc0611143bfd95: Status 404 returned error can't find the container with id 589b56f20362d3a31c2794396430824d3df1036b97e2b27a03bc0611143bfd95 Oct 30 19:50:13.260495 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.260469 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 19:50:13.262952 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.262917 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.263046 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.262991 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.263046 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.263020 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.519523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.519485 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" event={"ID":"0bf3d919-899c-4fcb-bbb8-704b70166626","Type":"ContainerStarted","Data":"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6"} Oct 30 19:50:13.519738 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.519594 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" event={"ID":"0bf3d919-899c-4fcb-bbb8-704b70166626","Type":"ContainerStarted","Data":"589b56f20362d3a31c2794396430824d3df1036b97e2b27a03bc0611143bfd95"} Oct 30 19:50:13.540880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.540847 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:50:13.545723 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:13.545679 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aa5c56_8fda_4878_9893_04074799918d.slice/crio-9d9c61e6a493f1cbbc5082a6402206f87cb2d270d0928f9db62a0b49ac103e56 WatchSource:0}: Error finding container 9d9c61e6a493f1cbbc5082a6402206f87cb2d270d0928f9db62a0b49ac103e56: Status 404 returned error can't find the container with id 9d9c61e6a493f1cbbc5082a6402206f87cb2d270d0928f9db62a0b49ac103e56 Oct 30 19:50:13.549552 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.549508 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.549703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.549601 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.549703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.549649 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:13.567747 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.566647 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podStartSLOduration=1.5666275280000002 podStartE2EDuration="1.566627528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.56396526 +0000 UTC m=+881.013457986" watchObservedRunningTime="2026-10-30 19:50:13.566627528 +0000 UTC m=+881.016120250" Oct 30 19:50:13.939129 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:13.939032 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:14.528516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:14.528470 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" event={"ID":"97aa5c56-8fda-4878-9893-04074799918d","Type":"ContainerStarted","Data":"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9"} Oct 30 19:50:14.528729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:14.528552 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" event={"ID":"97aa5c56-8fda-4878-9893-04074799918d","Type":"ContainerStarted","Data":"9d9c61e6a493f1cbbc5082a6402206f87cb2d270d0928f9db62a0b49ac103e56"} Oct 30 19:50:14.614389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:14.613973 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podStartSLOduration=2.6139507379999998 podStartE2EDuration="2.613950738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:14.583897108 +0000 UTC m=+882.033389856" watchObservedRunningTime="2026-10-30 19:50:14.613950738 +0000 UTC m=+882.063443464" Oct 30 19:50:14.940838 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:14.940749 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:14.941325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:14.940900 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:15.224642 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:15.223061 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:15.226467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:15.226434 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:15.226612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:15.226490 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:15.938300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:15.938258 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:15.938481 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:15.938315 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:16.222728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:16.222287 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:16.222728 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:16.222349 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:16.939617 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:16.939565 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:16.939809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:16.939641 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:17.225701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:17.225170 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:17.225701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:17.225242 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:17.938802 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:17.938768 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:17.938994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:17.938819 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:18.226835 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.226805 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:18.227489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.227440 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:18.229293 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.229265 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 19:50:18.235403 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.235354 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.253846 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.253804 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 19:50:18.260857 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.260826 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"openshift-service-ca.crt\"" Oct 30 19:50:18.261296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.261219 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"distinct-w5-6c4dd1-kserve-self-signed-certs\"" Oct 30 19:50:18.261465 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.261253 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"default-dockercfg-5xd58\"" Oct 30 19:50:18.261560 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.261431 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"kube-root-ca.crt\"" Oct 30 19:50:18.318796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.318765 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.319005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.318955 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.319005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.318994 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.319128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.319019 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbfc\" (UniqueName: \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.319128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.319054 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.319128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.319112 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421040 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421000 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421071 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421092 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421110 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbfc\" (UniqueName: \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421131 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421161 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421615 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421595 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.421780 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.421766 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.422507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.422482 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.424131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.424102 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.424285 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.424263 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.440239 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.440203 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbfc\" (UniqueName: \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\") pod \"distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.561260 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.561163 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:18.938912 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.938807 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:18.938912 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:18.938868 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:19.016440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.016346 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 19:50:19.019453 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:19.019122 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bac318_6c9d_4df0_97fa_8a123eddda8c.slice/crio-901dddfc6d95016dd5031dd93e7aa322a5a5f9e30584e7b6d931e5ace0c6ad6e WatchSource:0}: Error finding container 901dddfc6d95016dd5031dd93e7aa322a5a5f9e30584e7b6d931e5ace0c6ad6e: Status 404 returned error can't find the container with id 901dddfc6d95016dd5031dd93e7aa322a5a5f9e30584e7b6d931e5ace0c6ad6e Oct 30 19:50:19.225789 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.225751 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:19.226003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.225818 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:19.641206 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.641116 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" event={"ID":"68bac318-6c9d-4df0-97fa-8a123eddda8c","Type":"ContainerStarted","Data":"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734"} Oct 30 19:50:19.641206 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.641166 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" event={"ID":"68bac318-6c9d-4df0-97fa-8a123eddda8c","Type":"ContainerStarted","Data":"901dddfc6d95016dd5031dd93e7aa322a5a5f9e30584e7b6d931e5ace0c6ad6e"} Oct 30 19:50:19.678525 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.678457 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podStartSLOduration=1.678438577 podStartE2EDuration="1.678438577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:19.677800599 +0000 UTC m=+887.127293324" watchObservedRunningTime="2026-10-30 19:50:19.678438577 +0000 UTC m=+887.127931303" Oct 30 19:50:19.939359 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.939257 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:50:19.939359 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:19.939341 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:50:20.223302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.223263 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:50:20.223510 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.223331 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:50:20.562859 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.562732 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:20.871598 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:50:20.871501 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:50:20.943250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.943216 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:20.943623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.943604 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:20.944243 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:20.944226 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 19:50:21.227765 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:21.227734 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:21.228137 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:21.228112 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:21.229000 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:21.228980 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:50:23.781802 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:23.781764 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:50:27.037909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.037867 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 19:50:27.044292 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.044263 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.083746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.083709 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 19:50:27.185872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.185835 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.185872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.185875 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdvd\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186093 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.185947 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186093 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186090 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186112 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186234 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186222 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186271 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186240 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186275 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.186402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.186356 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.287763 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.287829 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.287859 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdvd\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.287900 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.288018 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.288045 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.288167 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.288187 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.288214 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.289034 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.289297 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.289640 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.289882 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.290174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.290130 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.314338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.308975 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.314338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.309200 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.317973 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.317936 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.323937 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.323907 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdvd\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.383849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.383811 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:27.704985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.704956 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 19:50:27.709709 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:27.709675 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a067728_9ddf_48d5_a8f5_04ee189986b3.slice/crio-918fe15f28bf6f8a9c8229e18823c4b44eb15cdd78c51a52e49be43a0d22c8af WatchSource:0}: Error finding container 918fe15f28bf6f8a9c8229e18823c4b44eb15cdd78c51a52e49be43a0d22c8af: Status 404 returned error can't find the container with id 918fe15f28bf6f8a9c8229e18823c4b44eb15cdd78c51a52e49be43a0d22c8af Oct 30 19:50:27.712716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.712678 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:27.712818 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.712746 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:27.712818 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.712775 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:27.728215 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:27.728184 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" event={"ID":"8a067728-9ddf-48d5-a8f5-04ee189986b3","Type":"ContainerStarted","Data":"918fe15f28bf6f8a9c8229e18823c4b44eb15cdd78c51a52e49be43a0d22c8af"} Oct 30 19:50:28.562934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:28.562896 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:28.605382 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:28.605333 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:28.736393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:28.736327 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" event={"ID":"8a067728-9ddf-48d5-a8f5-04ee189986b3","Type":"ContainerStarted","Data":"bef6eab063e7ed941c28d35e4f22dcc182a87335e44591305b27f608319b6966"} Oct 30 19:50:28.747774 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:28.747746 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:28.790195 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:28.789852 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podStartSLOduration=2.789832203 podStartE2EDuration="2.789832203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.783941346 +0000 UTC m=+896.233434072" watchObservedRunningTime="2026-10-30 19:50:28.789832203 +0000 UTC m=+896.239324932" Oct 30 19:50:29.388622 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:29.388571 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:29.390422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:29.390391 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:29.390552 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:29.390449 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:30.388840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:30.388804 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.389254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:30.388864 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:31.385012 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:31.384971 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.385315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:31.385282 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:32.390503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:32.389531 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.390503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:32.389622 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:32.840047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:32.840015 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:50:32.846679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:32.845576 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:50:33.388794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:33.388759 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.388981 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:33.388820 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:34.385000 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:34.384962 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.385442 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:34.385017 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:50:35.389746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:35.389711 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:35.390250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:35.390120 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:35.390849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:35.390827 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 19:50:42.013599 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.013558 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:50:42.019437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.019411 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.056866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.056829 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:50:42.120484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120451 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4k7n\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120497 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120517 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120653 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120671 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120687 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120706 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120802 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.120992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.120899 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222467 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222505 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222522 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222540 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222574 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222629 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222973 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222843 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4k7n\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222973 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222907 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.222973 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222939 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.223117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.222986 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.223117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.223063 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.223219 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.223115 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.223274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.223258 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.223378 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.223348 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.224785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.224757 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.225418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.225402 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.247482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.247450 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.247627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.247536 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4k7n\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.291423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.291326 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:50:42.298375 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.298335 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.327429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.327393 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:50:42.335841 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.335811 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:42.426296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426252 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426346 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbgg\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426390 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426512 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426584 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426676 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426632 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426693 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426780 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426742 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.426891 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.426776 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528750 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528825 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528853 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528899 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528946 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.528989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.528980 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529081 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529147 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbgg\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529194 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529288 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529401 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529454 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529616 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529565 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.529926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.529904 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.531354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.531329 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.531574 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.531552 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.541633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.541572 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:50:42.547327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.547310 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.563095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.559246 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.563095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.560888 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbgg\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.576474 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.576428 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:50:42.618840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.618797 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:42.705737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.704968 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:42.705737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.705083 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:42.705737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.705127 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:42.705737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.705521 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:50:42.731840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.731811 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.731992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.731859 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.731992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.731963 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.731992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.731982 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.732106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.732041 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.732106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.732087 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.732183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.732109 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.732183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.732129 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4fj\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.732246 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.732187 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833284 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833190 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833315 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833350 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833452 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833520 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833556 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833582 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4fj\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833679 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833669 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833670 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833810 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833816 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.833898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.833882 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.834103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.834059 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.834406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.834356 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.836312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.836290 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.836955 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.836935 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.864552 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.864511 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.865014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.864990 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4fj\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.866881 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.866852 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:42.868143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.868113 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" event={"ID":"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21","Type":"ContainerStarted","Data":"8ad4b9ed7116c4b088fa3aaa9e7a7d0be5a1e9cc217ca3398945c5aea10952b5"} Oct 30 19:50:42.868227 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.868173 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" event={"ID":"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21","Type":"ContainerStarted","Data":"b19e1a6934dab7f88603447f04c21f83235563d041e48ec2121e2adccb2b9615"} Oct 30 19:50:42.928358 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.927516 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podStartSLOduration=1.927495506 podStartE2EDuration="1.927495506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:42.923144038 +0000 UTC m=+910.372636769" watchObservedRunningTime="2026-10-30 19:50:42.927495506 +0000 UTC m=+910.376988234" Oct 30 19:50:42.995502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.995464 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:50:42.998209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.998104 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:42.998209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.998181 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:42.998582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:42.998483 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:43.206794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.206753 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:50:43.210917 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:43.210887 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854bf2a9_4868_4b62_a01e_71c85911e4ff.slice/crio-1d302e0a0610b97e41442fc9ce14fa0a8e7b7dd6256db243217fbed45c5f481c WatchSource:0}: Error finding container 1d302e0a0610b97e41442fc9ce14fa0a8e7b7dd6256db243217fbed45c5f481c: Status 404 returned error can't find the container with id 1d302e0a0610b97e41442fc9ce14fa0a8e7b7dd6256db243217fbed45c5f481c Oct 30 19:50:43.213854 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.213809 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:43.213952 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.213905 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:43.213992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.213951 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:43.336641 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.336564 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:43.338490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.338459 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:43.338611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.338519 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:43.917985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.917946 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" event={"ID":"0e07c839-ca46-4f3e-9301-0d86384e4dc3","Type":"ContainerStarted","Data":"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01"} Oct 30 19:50:43.918313 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.918275 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" event={"ID":"0e07c839-ca46-4f3e-9301-0d86384e4dc3","Type":"ContainerStarted","Data":"f312ab9e0d0134fe14ce251e5f540e77ea6474d6c05797388ac936ec4fecbd32"} Oct 30 19:50:43.923529 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.923494 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" event={"ID":"854bf2a9-4868-4b62-a01e-71c85911e4ff","Type":"ContainerStarted","Data":"a353b2118a782d482401de021b16555b1f4f9a6720e71a6e77bb12ee373ec0f6"} Oct 30 19:50:43.923777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:43.923751 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" event={"ID":"854bf2a9-4868-4b62-a01e-71c85911e4ff","Type":"ContainerStarted","Data":"1d302e0a0610b97e41442fc9ce14fa0a8e7b7dd6256db243217fbed45c5f481c"} Oct 30 19:50:44.045942 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.045236 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podStartSLOduration=2.045216231 podStartE2EDuration="2.045216231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.988699551 +0000 UTC m=+911.438192277" watchObservedRunningTime="2026-10-30 19:50:44.045216231 +0000 UTC m=+911.494708956" Oct 30 19:50:44.048292 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.048243 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podStartSLOduration=2.04823004 podStartE2EDuration="2.04823004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:44.042308859 +0000 UTC m=+911.491801596" watchObservedRunningTime="2026-10-30 19:50:44.04823004 +0000 UTC m=+911.497722764" Oct 30 19:50:44.340390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.338826 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:44.340390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.338889 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:44.619965 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.619867 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:44.640388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.634513 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:44.640388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.634583 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:44.868387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.868290 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:44.871234 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.871109 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:44.871555 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:44.871524 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:45.342192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.342130 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:45.342832 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.342774 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:45.641461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.640505 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:45.641461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.640588 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:45.875410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.869159 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:45.875410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:45.869243 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:46.337849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.337811 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:46.338191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.338162 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:46.641475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.623067 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:46.641475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.623133 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:46.871388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.867839 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:46.871388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:46.867903 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:47.344394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.340778 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:47.344394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.340844 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:47.627630 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.627534 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:47.627879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.627851 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:47.869885 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.869842 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:47.870542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:47.870506 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:48.326512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.326465 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:50:48.336823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.336781 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:48.337040 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.336848 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:48.369268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.369223 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:50:48.592450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.591469 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.623389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.622874 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:48.623389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.622955 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:48.744391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.742897 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.868055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.867942 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:48.868329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:48.868300 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:49.194667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.194562 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:49.359424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.330961 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:50:49.359424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.331505 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:50:49.533462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.438698 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:49.533462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.438777 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:49.533462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.516390 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:50:49.579390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.570815 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:50:49.584269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.584225 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:49.632959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.628817 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.001424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.630073 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:50:50.178335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.661907 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.178700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.178667 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:50.179261 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.179231 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:50:50.181250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:49.759729 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.186324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.186287 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:50.313841 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.313726 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:50:50.359626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.359585 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:50:50.512254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.512196 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.582463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.574323 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:50.582463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.573608 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.582463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.574465 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:50.582463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.577968 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.711389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.680384 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:50:50.769648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.747649 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.794513 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.794454 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:50.965986 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:50.965882 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:50:51.084321 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.084274 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:50:51.253014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.226543 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:50:51.348427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.340709 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:51.552447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.525569 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:51.552447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.525632 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:51.552447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.526840 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:51.552447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.526886 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:51.593472 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.593423 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:51.621069 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.621034 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:51.643389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.621285 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:51.759965 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:50:51.759927 5991 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.013s" Oct 30 19:50:51.760135 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.760046 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:50:51.814515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.782546 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:51.851103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.851070 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:50:51.880985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.880942 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:51.881265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.881235 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:51.979388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:51.969614 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:52.019234 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:50:52.014082 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:50:52.026708 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.025405 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:50:52.035657 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.035631 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:50:52.049416 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.038927 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:50:52.049818 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.049758 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:50:52.068420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.068311 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:50:52.336446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.336311 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:52.336446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.336396 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:50:52.621256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.621163 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:52.621256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.621207 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:50:52.868389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.867953 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:50:52.868389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:52.868020 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:50:53.353835 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.353790 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:53.354182 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.354158 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:53.355200 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.355176 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:50:53.625622 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.625510 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:53.626113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.626071 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:53.627300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.627270 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:50:53.872526 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.872496 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:53.872809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.872787 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:53.873467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:53.873443 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:50:57.495835 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.495802 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 19:50:57.500311 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.500285 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.530877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.530843 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 19:50:57.602288 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602246 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4bs\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602295 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602497 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602520 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602541 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602600 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602667 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602701 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.602851 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.602803 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704244 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704211 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704244 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704246 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704270 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704301 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704378 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704696 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704553 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704696 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704679 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704720 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704742 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704735 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4bs\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.704966 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.704858 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.705112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.705090 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.705181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.705102 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.705420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.705399 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.707138 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.707113 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.707417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.707398 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.722635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.722599 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.722824 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.722805 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4bs\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:57.815596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:57.815498 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:58.128609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.128198 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:50:58.133855 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.133821 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.162231 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:58.162193 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0949fc_09a7_450b_94c8_6c4df5c323a7.slice/crio-b6fa4aa9b59de5aed095cb23cba97c5306dc4d2651c55e8a9bf0fafb189adf8c WatchSource:0}: Error finding container b6fa4aa9b59de5aed095cb23cba97c5306dc4d2651c55e8a9bf0fafb189adf8c: Status 404 returned error can't find the container with id b6fa4aa9b59de5aed095cb23cba97c5306dc4d2651c55e8a9bf0fafb189adf8c Oct 30 19:50:58.162600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.162530 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 19:50:58.168723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.167112 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:50:58.170661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.170611 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.170797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.170695 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.170797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.170742 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.314090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314054 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314097 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314174 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314193 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314227 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314263 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314314 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9svk\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314406 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.314718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.314461 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.415805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.415713 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.415805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.415754 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.415805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.415772 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9svk\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.415805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.415804 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.415854 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416038 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416065 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416110 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416130 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416436 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416338 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416497 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416431 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416547 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416618 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416598 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.416908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.416877 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.418466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.418447 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.418466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.418454 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.433734 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.433693 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.433875 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.433784 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9svk\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.453460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.453413 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:50:58.783213 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.783181 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:50:58.785620 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:50:58.785589 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71904f1_8891_4579_ba29_dd657af39ccb.slice/crio-294ec602cc6b35f9ebe8f8fafbf2ee1aa362fe6c981fb8e679bf3c49fd0f3e73 WatchSource:0}: Error finding container 294ec602cc6b35f9ebe8f8fafbf2ee1aa362fe6c981fb8e679bf3c49fd0f3e73: Status 404 returned error can't find the container with id 294ec602cc6b35f9ebe8f8fafbf2ee1aa362fe6c981fb8e679bf3c49fd0f3e73 Oct 30 19:50:58.788908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.788868 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.789012 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.788956 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.789012 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.788998 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:58.813558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.813510 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" event={"ID":"9f0949fc-09a7-450b-94c8-6c4df5c323a7","Type":"ContainerStarted","Data":"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf"} Oct 30 19:50:58.813749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.813623 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" event={"ID":"9f0949fc-09a7-450b-94c8-6c4df5c323a7","Type":"ContainerStarted","Data":"b6fa4aa9b59de5aed095cb23cba97c5306dc4d2651c55e8a9bf0fafb189adf8c"} Oct 30 19:50:58.821818 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.821775 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:50:58.823111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.823078 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" event={"ID":"f71904f1-8891-4579-ba29-dd657af39ccb","Type":"ContainerStarted","Data":"294ec602cc6b35f9ebe8f8fafbf2ee1aa362fe6c981fb8e679bf3c49fd0f3e73"} Oct 30 19:50:58.859070 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:58.859019 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podStartSLOduration=1.8590057930000001 podStartE2EDuration="1.859005793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:58.857653502 +0000 UTC m=+926.307146254" watchObservedRunningTime="2026-10-30 19:50:58.859005793 +0000 UTC m=+926.308498517" Oct 30 19:50:59.818090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:59.818047 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:59.818513 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:59.818101 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:50:59.831338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:50:59.831304 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" event={"ID":"f71904f1-8891-4579-ba29-dd657af39ccb","Type":"ContainerStarted","Data":"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f"} Oct 30 19:51:00.454758 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:00.454723 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:51:00.456415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:00.456385 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:00.456552 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:00.456437 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:00.816387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:00.815937 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:51:00.816387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:00.815996 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:51:01.454327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:01.454289 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:01.454796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:01.454360 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:01.818484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:01.818442 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:51:01.818799 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:01.818764 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:51:02.454546 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:02.454500 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:02.454994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:02.454568 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:02.817388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:02.816986 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:51:02.817388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:02.817069 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:51:03.454355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:03.454320 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:03.454561 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:03.454392 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:03.816389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:03.816147 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:51:03.816389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:03.816227 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:51:04.455111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:04.455074 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:04.455334 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:04.455135 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:04.816687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:04.816642 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:51:04.817129 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:04.816717 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:51:05.454986 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.454949 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:51:05.455179 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.455014 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:51:05.820341 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.820306 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:51:05.820864 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.820848 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:51:05.821639 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.821614 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 19:51:05.872181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:05.872107 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podStartSLOduration=7.872088584 podStartE2EDuration="7.872088584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:59.885860016 +0000 UTC m=+927.335352762" watchObservedRunningTime="2026-10-30 19:51:05.872088584 +0000 UTC m=+933.321581307" Oct 30 19:51:06.484015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:06.483981 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:51:06.484939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:06.484897 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:51:06.485785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:06.485765 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:51:08.574948 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:08.574913 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:51:08.575389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:08.575136 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" containerID="cri-o://c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c" gracePeriod=30 Oct 30 19:51:08.584925 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:08.584893 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:51:08.585132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:08.585097 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" containerID="cri-o://3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73" gracePeriod=30 Oct 30 19:51:09.972907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:09.972859 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 19:51:09.978881 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:09.978854 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.005906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.005855 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"distinct-2-w3-01f7d3-kserve-self-signed-certs\"" Oct 30 19:51:10.005906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.005886 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"openshift-service-ca.crt\"" Oct 30 19:51:10.005906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.005913 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"kube-root-ca.crt\"" Oct 30 19:51:10.006207 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.005944 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"default-dockercfg-8cntf\"" Oct 30 19:51:10.010487 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.010448 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 19:51:10.065328 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065294 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.065328 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065334 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.065602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065467 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjjk\" (UniqueName: \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.065602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065523 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.065602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065570 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.065753 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.065732 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.181829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.181787 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.181963 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.182002 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182175 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.182061 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjjk\" (UniqueName: \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182175 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.182131 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182271 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.182185 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.182790 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.182762 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.183704 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.183678 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.183939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.183915 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.193149 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.193112 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.220420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.220201 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.229190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.229113 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjjk\" (UniqueName: \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\") pod \"distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.292313 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.292265 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:10.602448 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.602415 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 19:51:10.603424 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:51:10.603394 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6207f6_765e_4cb5_a8de_83106ab16872.slice/crio-bf6a17b2e2506f45d3a9b5a05681ca1387eeaa9239a4856c969c39581d936104 WatchSource:0}: Error finding container bf6a17b2e2506f45d3a9b5a05681ca1387eeaa9239a4856c969c39581d936104: Status 404 returned error can't find the container with id bf6a17b2e2506f45d3a9b5a05681ca1387eeaa9239a4856c969c39581d936104 Oct 30 19:51:10.925626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.925527 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" event={"ID":"ce6207f6-765e-4cb5-a8de-83106ab16872","Type":"ContainerStarted","Data":"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26"} Oct 30 19:51:10.925626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:10.925575 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" event={"ID":"ce6207f6-765e-4cb5-a8de-83106ab16872","Type":"ContainerStarted","Data":"bf6a17b2e2506f45d3a9b5a05681ca1387eeaa9239a4856c969c39581d936104"} Oct 30 19:51:11.006035 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:11.005966 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podStartSLOduration=2.005942911 podStartE2EDuration="2.005942911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:11.004306747 +0000 UTC m=+938.453799471" watchObservedRunningTime="2026-10-30 19:51:11.005942911 +0000 UTC m=+938.455435635" Oct 30 19:51:11.293183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:11.293149 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:12.892097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:12.892056 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 19:51:12.896608 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:12.896579 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:12.934184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:12.934150 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 19:51:13.026099 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026059 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026280 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026133 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026283 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026342 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026501 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026478 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026510 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026601 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026547 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026694 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026677 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.026756 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.026738 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbxm\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.128831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.128789 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.128862 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.128900 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129120 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129091 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129166 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129248 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129233 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129308 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129270 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129308 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129294 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbxm\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129319 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129398 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129586 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129682 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129869 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.129919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.129870 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.131768 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.131746 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.132043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.132024 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.171164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.171078 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.171164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.171088 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbxm\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.212891 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.212848 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:13.542589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.542557 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 19:51:13.546553 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.546515 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:13.546661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.546598 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:13.546661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.546640 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:13.865518 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.865477 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:51:13.931740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.931711 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:51:13.952947 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.952914 5991 generic.go:356] "Generic (PLEG): container finished" podID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerID="c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c" exitCode=0 Oct 30 19:51:13.953148 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.952959 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" event={"ID":"d8ece28d-bb64-4681-8ce1-a68619cec5da","Type":"ContainerDied","Data":"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c"} Oct 30 19:51:13.953148 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.953029 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" Oct 30 19:51:13.953340 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.953320 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb" event={"ID":"d8ece28d-bb64-4681-8ce1-a68619cec5da","Type":"ContainerDied","Data":"ad7eaa07c5d2da1146fe8bb6db46e8f2492f4d0837e0858245d721ef19586399"} Oct 30 19:51:13.953503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.953397 5991 scope.go:122] "RemoveContainer" containerID="c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c" Oct 30 19:51:13.955630 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.955515 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" event={"ID":"6dbea309-e025-4968-b02c-f0b5c6913317","Type":"ContainerStarted","Data":"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617"} Oct 30 19:51:13.955630 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.955569 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" event={"ID":"6dbea309-e025-4968-b02c-f0b5c6913317","Type":"ContainerStarted","Data":"95479ef14e35f36bb89b799197488715fe56c8c7f998c87b1a53a40ef3de07c4"} Oct 30 19:51:13.957148 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.957116 5991 generic.go:356] "Generic (PLEG): container finished" podID="51753c8d-594b-480e-8a26-6fc135c47696" containerID="3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73" exitCode=0 Oct 30 19:51:13.957254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.957210 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" event={"ID":"51753c8d-594b-480e-8a26-6fc135c47696","Type":"ContainerDied","Data":"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73"} Oct 30 19:51:13.957254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.957248 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" event={"ID":"51753c8d-594b-480e-8a26-6fc135c47696","Type":"ContainerDied","Data":"81e184b73562559fc9ebebbc34588431fe73aabaac844d4e4c7850471fbeb65c"} Oct 30 19:51:13.957383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.957318 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4" Oct 30 19:51:13.964667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.964632 5991 scope.go:122] "RemoveContainer" containerID="c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c" Oct 30 19:51:13.965023 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:51:13.964998 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c\": container with ID starting with c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c not found: ID does not exist" containerID="c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c" Oct 30 19:51:13.965135 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.965038 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c"} err="failed to get container status \"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c\": rpc error: code = NotFound desc = could not find container \"c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c\": container with ID starting with c8f9917b9c8a9e47b205f867ed8217fa9fa55250e35e0014b4000d3a563eaa4c not found: ID does not exist" Oct 30 19:51:13.965135 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.965063 5991 scope.go:122] "RemoveContainer" containerID="3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73" Oct 30 19:51:13.985601 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.985570 5991 scope.go:122] "RemoveContainer" containerID="3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73" Oct 30 19:51:13.986641 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:51:13.986519 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73\": container with ID starting with 3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73 not found: ID does not exist" containerID="3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73" Oct 30 19:51:13.986864 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:13.986815 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73"} err="failed to get container status \"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73\": rpc error: code = NotFound desc = could not find container \"3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73\": container with ID starting with 3b5790c05475c8676816f463c30597cb45360c20c028abf7706a8356483c8f73 not found: ID does not exist" Oct 30 19:51:14.029055 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.028994 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podStartSLOduration=2.028971557 podStartE2EDuration="2.028971557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.026294229 +0000 UTC m=+941.475786955" watchObservedRunningTime="2026-10-30 19:51:14.028971557 +0000 UTC m=+941.478464285" Oct 30 19:51:14.051796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.051754 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.052031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.051818 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.052106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052025 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.052106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052093 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.052709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052641 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.052709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052683 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.052896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052725 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.052896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052760 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.052896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052793 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.052896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052862 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.053103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052908 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.053103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.052934 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.053103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053072 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.053296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053136 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.053296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053227 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.053296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\") pod \"d8ece28d-bb64-4681-8ce1-a68619cec5da\" (UID: \"d8ece28d-bb64-4681-8ce1-a68619cec5da\") " Oct 30 19:51:14.053466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053346 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.053466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053414 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\") pod \"51753c8d-594b-480e-8a26-6fc135c47696\" (UID: \"51753c8d-594b-480e-8a26-6fc135c47696\") " Oct 30 19:51:14.054427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053908 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.054427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.054089 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.054427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.054342 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.054427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.054379 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.054427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.054341 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:14.055467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.054830 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:14.055467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.055003 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.055467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.055046 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.055467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.053916 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.055467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.055408 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.056138 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.055736 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.056420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.056395 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.056637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.056614 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:14.057152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.057115 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:14.057316 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.057225 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.057976 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.057953 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r" pod "51753c8d-594b-480e-8a26-6fc135c47696" (UID: "51753c8d-594b-480e-8a26-6fc135c47696"). InnerVolumeSpecName "kube-api-access-w5t8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:14.058603 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.058583 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:14.058915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.058834 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:14.059712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.059688 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:14.060236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.060212 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx" pod "d8ece28d-bb64-4681-8ce1-a68619cec5da" (UID: "d8ece28d-bb64-4681-8ce1-a68619cec5da"). InnerVolumeSpecName "kube-api-access-w7xhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:14.155512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155480 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ece28d-bb64-4681-8ce1-a68619cec5da-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155509 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xhx\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-kube-api-access-w7xhx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155523 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155537 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5t8r\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-kube-api-access-w5t8r\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155549 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155559 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51753c8d-594b-480e-8a26-6fc135c47696-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155569 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155580 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ece28d-bb64-4681-8ce1-a68619cec5da-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155590 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155601 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51753c8d-594b-480e-8a26-6fc135c47696-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155612 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155625 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155635 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51753c8d-594b-480e-8a26-6fc135c47696-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155645 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ece28d-bb64-4681-8ce1-a68619cec5da-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155656 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.155727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.155670 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51753c8d-594b-480e-8a26-6fc135c47696-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:51:14.214350 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.214320 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:14.215987 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.215956 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:14.216131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.216036 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:14.297105 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.297067 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:51:14.302574 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.302543 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-jt5xb"] Oct 30 19:51:14.338335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.338242 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:51:14.341227 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.341200 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wscc4"] Oct 30 19:51:14.712878 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.712796 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51753c8d-594b-480e-8a26-6fc135c47696" path="/var/lib/kubelet/pods/51753c8d-594b-480e-8a26-6fc135c47696/volumes" Oct 30 19:51:14.713352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:14.713339 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" path="/var/lib/kubelet/pods/d8ece28d-bb64-4681-8ce1-a68619cec5da/volumes" Oct 30 19:51:15.213913 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:15.213875 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.214292 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:15.213936 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:16.214654 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:16.214619 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.215053 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:16.214672 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:17.214248 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:17.214213 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.214536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:17.214268 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:18.214249 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:18.214216 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.214668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:18.214269 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:19.127662 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:19.127626 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:51:19.213616 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:19.213582 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.213807 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:19.213632 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:20.214111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.214070 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.214573 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.214150 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:51:20.292654 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.292616 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:20.307911 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.307876 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.337642 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338565 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338581 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338621 5991 cpu_manager.go:414] "containerMap: removing container" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338626 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338714 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ece28d-bb64-4681-8ce1-a68619cec5da" containerName="istio-proxy" Oct 30 19:51:20.340463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.338728 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="51753c8d-594b-480e-8a26-6fc135c47696" containerName="istio-proxy" Oct 30 19:51:20.342515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.342488 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.357773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.357738 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"default-dockercfg-ppjjf\"" Oct 30 19:51:20.360377 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.360347 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"distinct-2-w2-284bf6-kserve-self-signed-certs\"" Oct 30 19:51:20.361553 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.361529 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"kube-root-ca.crt\"" Oct 30 19:51:20.361553 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.361541 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"openshift-service-ca.crt\"" Oct 30 19:51:20.364872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.364849 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 19:51:20.470252 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470156 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.470252 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470208 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7nf\" (UniqueName: \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.470509 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470297 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.470509 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470405 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.470509 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470455 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.470627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.470571 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.571773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.571729 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7nf\" (UniqueName: \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.571970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.571783 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.571970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.571840 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.571970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.571898 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.572146 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.572004 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.572268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.572251 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.572415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.572391 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.572493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.572415 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.572598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.572580 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.574409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.574384 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.574670 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.574653 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.593409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.591155 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7nf\" (UniqueName: \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\") pod \"distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.660942 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:20.659681 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:20.934646 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:51:20.934572 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:51:21.078425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:21.072883 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 19:51:21.078425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:21.078076 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:21.483057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:21.483025 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:21.483611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:21.483415 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:21.484582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:21.484559 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 19:51:22.046825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:22.046787 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" event={"ID":"e67103ec-b3dd-4fc5-892b-1c1a9b113b07","Type":"ContainerStarted","Data":"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b"} Oct 30 19:51:22.047037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:22.046836 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" event={"ID":"e67103ec-b3dd-4fc5-892b-1c1a9b113b07","Type":"ContainerStarted","Data":"0f6f2d900f42c2caf4b71c77ec8f778c73b43e81ea1c58078065638dd5be7c83"} Oct 30 19:51:22.097249 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:22.092873 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podStartSLOduration=2.092855466 podStartE2EDuration="2.092855466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:22.091972292 +0000 UTC m=+949.541465028" watchObservedRunningTime="2026-10-30 19:51:22.092855466 +0000 UTC m=+949.542348194" Oct 30 19:51:22.660427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:22.660394 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:28.362173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.362137 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 19:51:28.367117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.367086 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.390545 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.390509 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 19:51:28.395989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.395955 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"distinct-2-w5-6c4dd1-kserve-self-signed-certs\"" Oct 30 19:51:28.530170 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530126 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.530401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530256 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.530401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530302 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqqmq\" (UniqueName: \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.530401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530358 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.530529 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530502 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.530650 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.530628 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.633507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.633331 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.634231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.634207 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqqmq\" (UniqueName: \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.642249 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.642214 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.642774 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.642752 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.643236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643223 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.643702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643687 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.644131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643006 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.646632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643581 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.646792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643077 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.646913 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.643993 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.648413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.648382 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.654075 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.654041 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqqmq\" (UniqueName: \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\") pod \"distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:28.688536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:28.688465 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:29.015083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:29.015049 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 19:51:29.016646 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:51:29.016618 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf8af32d_b630_45ce_93e9_a1df20cff8b3.slice/crio-71aa6825d4352cf1c1f91e039256086fad8b7e56d9b96c913a7b2f24d2a53d65 WatchSource:0}: Error finding container 71aa6825d4352cf1c1f91e039256086fad8b7e56d9b96c913a7b2f24d2a53d65: Status 404 returned error can't find the container with id 71aa6825d4352cf1c1f91e039256086fad8b7e56d9b96c913a7b2f24d2a53d65 Oct 30 19:51:29.135667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:29.135629 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" event={"ID":"af8af32d-b630-45ce-93e9-a1df20cff8b3","Type":"ContainerStarted","Data":"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963"} Oct 30 19:51:29.135667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:29.135672 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" event={"ID":"af8af32d-b630-45ce-93e9-a1df20cff8b3","Type":"ContainerStarted","Data":"71aa6825d4352cf1c1f91e039256086fad8b7e56d9b96c913a7b2f24d2a53d65"} Oct 30 19:51:29.177269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:29.177198 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podStartSLOduration=1.177178991 podStartE2EDuration="1.177178991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:29.172856109 +0000 UTC m=+956.622348834" watchObservedRunningTime="2026-10-30 19:51:29.177178991 +0000 UTC m=+956.626671717" Oct 30 19:51:29.689267 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:29.689223 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:30.660439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:30.660397 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:30.680142 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:30.680107 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:31.163740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:31.163708 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:38.689972 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:38.689936 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:38.703275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:38.703238 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:39.228602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:39.228573 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:42.235269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.235234 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 19:51:42.241468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.241441 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.272468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.272422 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 19:51:42.350905 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.350857 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351066 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.350930 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351066 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351038 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351170 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351104 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351243 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351219 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351331 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351448 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351572 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351557 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.351619 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.351604 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9rw\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.453582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453545 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.453791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453663 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.453791 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453761 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.453903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453796 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454016 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453911 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454016 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453957 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9rw\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454016 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.453990 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.454045 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.454088 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454304 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.454255 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.454562 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.454525 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.455011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.454987 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.455226 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.455029 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.455938 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.455916 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.456826 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.456800 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.457638 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.457617 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.473067 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.473032 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.473214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.473159 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9rw\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.556895 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.556790 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:42.882594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.881979 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 19:51:42.882753 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:51:42.882711 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f923218_9ec5_4f7f_bee7_c303477e64ab.slice/crio-6a4dab957549be9c601d73e781f21197d1609e085f8d95a5068079f8c83f4bea WatchSource:0}: Error finding container 6a4dab957549be9c601d73e781f21197d1609e085f8d95a5068079f8c83f4bea: Status 404 returned error can't find the container with id 6a4dab957549be9c601d73e781f21197d1609e085f8d95a5068079f8c83f4bea Oct 30 19:51:42.892152 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.891876 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:42.892466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.892436 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:42.892567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:42.892511 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:51:43.252091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.252055 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" event={"ID":"2f923218-9ec5-4f7f-bee7-c303477e64ab","Type":"ContainerStarted","Data":"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71"} Oct 30 19:51:43.252632 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.252115 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" event={"ID":"2f923218-9ec5-4f7f-bee7-c303477e64ab","Type":"ContainerStarted","Data":"6a4dab957549be9c601d73e781f21197d1609e085f8d95a5068079f8c83f4bea"} Oct 30 19:51:43.303629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.303560 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podStartSLOduration=1.30354286 podStartE2EDuration="1.30354286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:43.301248947 +0000 UTC m=+970.750741674" watchObservedRunningTime="2026-10-30 19:51:43.30354286 +0000 UTC m=+970.753035622" Oct 30 19:51:43.557357 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.557267 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:43.559133 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.559104 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:43.559272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:43.559155 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:44.558187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:44.558150 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:44.558734 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:44.558219 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:45.557726 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:45.557689 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:45.557913 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:45.557746 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:46.591446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:46.585337 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:46.591958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:46.591902 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:47.558045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:47.558011 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:47.558216 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:47.558069 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:48.557469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:48.557435 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:48.557967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:48.557501 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:49.558127 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:49.558084 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:49.558554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:49.558157 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:51:51.105755 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:51:51.105670 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:51:51.562191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.561489 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:51.562191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.561578 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.666576 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.666517 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.804514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.804465 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.849763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.849654 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:51.850053 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.850015 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.981553 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:51:51.990677 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.990494 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:51.990677 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:51.990592 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.030837 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.030788 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.041389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.040742 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.041389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.041163 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:51:52.079109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.079068 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.160581 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.160463 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.224896 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.224851 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.382908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.381634 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.617391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.593487 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:52.617391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.593583 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.617391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.594616 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:51:52.654656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.654464 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.675422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.660071 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:52.675422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:52.660227 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.047811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.043460 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.069192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.068079 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:51:53.069192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.068888 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:51:53.069192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.068956 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:51:53.102392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.071819 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.120176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.114824 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:53.120176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.090415 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.120631 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.120602 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:51:53.149872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.090909 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:51:53.476120 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.229495 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.502674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.265015 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:51:53.582390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.390319 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.582390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.500504 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:51:53.582390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.515596 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:53.582390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.515671 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:53.582390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.515728 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:51:53.594864 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.594818 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.595489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.595463 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:51:53.689770 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.687530 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:53.689770 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:53.687610 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.046389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.045643 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.077212 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.077168 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.195491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.195438 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.217643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.189136 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.425664 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.425471 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:54.425664 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.425536 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.472393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.466563 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.533400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.531889 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.533400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.532313 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.533400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.532612 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.622393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.599768 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.694007 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.693907 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:54.694269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.694237 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.805901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.731470 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:54.805901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:54.731542 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.137553 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.137478 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.237277 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.230955 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" podUID="62b90662-f37f-4ba0-a9fc-9b37bcdfe838" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 19:51:55.237277 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.232657 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:51:55.253390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.249914 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.515421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.502528 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.553544 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.553495 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:55.554187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.554154 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.616387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.583031 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:51:55.674391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.584509 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:51:55.674391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.584579 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:51:55.674391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.645033 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:51:55.674391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.664493 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.900518 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.818493 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:51:55.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.978481 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:55.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:55.978553 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.186475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.145801 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.298644 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.287582 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:51:56.357952 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.357896 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:51:56.523934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.523880 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.702467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.702420 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:51:56.713645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.713604 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:51:56.737532 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.737479 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.761056 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.761007 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.918240 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.916878 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:51:56.918240 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:56.918053 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.036821 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.036780 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:57.037121 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.037082 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.146688 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.146634 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.267447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.267410 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:51:57.267653 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.267469 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:51:57.300387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.298642 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.394418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.394386 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:51:57.396207 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.396186 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:51:57.397236 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.397219 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:51:57.428592 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.428559 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:51:57.434565 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.434536 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:51:57.435796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.435776 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:51:57.451427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.451401 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:51:57.454811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.454787 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:57.455879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.455861 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:51:57.456944 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:51:57.456924 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 19:52:12.264496 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.264453 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:52:12.269613 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.269582 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.282335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.282305 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:52:12.286757 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.286735 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.293114 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.293086 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:52:12.309863 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.309830 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:52:12.314090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314059 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314174 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8wp\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314206 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314275 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314298 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314348 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314416 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314481 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.314746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.314562 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.417193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417155 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mj64\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417305 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.417395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417384 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417438 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417486 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417541 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8wp\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.417745 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417606 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417745 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417697 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.417745 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417732 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417839 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.417906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417892 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417927 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417967 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.417997 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418026 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418068 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418084 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418117 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418150 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418450 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418681 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418656 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.418940 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.418918 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.419408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.419384 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.419930 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.419903 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.421065 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.421045 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.436110 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.436080 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.436464 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.436443 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8wp\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.519527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519443 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519552 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519662 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mj64\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519766 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519819 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519864 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519899 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.519964 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520004 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520023 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520134 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520161 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520628 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520401 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.520665 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.520638 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.522109 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.522084 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.522355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.522336 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.539235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.539202 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mj64\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.539406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.539244 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.594615 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.594579 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:12.604908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.604876 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:12.808830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.808747 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:52:12.814842 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.814822 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.873694 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.873654 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:52:12.907793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.907761 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:52:12.913484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.913451 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:12.930078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930042 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930228 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930185 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930228 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930221 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930271 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930474 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930454 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjk2w\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930628 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930612 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930657 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930765 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930741 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.930850 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.930834 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:12.972893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:12.972863 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:52:13.006572 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.006536 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:52:13.017530 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:52:13.017490 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de6513d_f9e2_4166_9390_7e8ac17ef3f5.slice/crio-b519a9fb4e19ea38e91eddcf4db01ee5076c3b9885522acaf460fe286eaba9f2 WatchSource:0}: Error finding container b519a9fb4e19ea38e91eddcf4db01ee5076c3b9885522acaf460fe286eaba9f2: Status 404 returned error can't find the container with id b519a9fb4e19ea38e91eddcf4db01ee5076c3b9885522acaf460fe286eaba9f2 Oct 30 19:52:13.019690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.019649 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:52:13.020976 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.020943 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.021058 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.021009 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.021058 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.021037 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.026494 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:52:13.026468 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fe2ac9_4d7f_405c_ac64_236c2166abfe.slice/crio-e3ba6061a3ec40b1e8c9f91074249ed7e316acc527dc35b3755c3f4f437d8ef1 WatchSource:0}: Error finding container e3ba6061a3ec40b1e8c9f91074249ed7e316acc527dc35b3755c3f4f437d8ef1: Status 404 returned error can't find the container with id e3ba6061a3ec40b1e8c9f91074249ed7e316acc527dc35b3755c3f4f437d8ef1 Oct 30 19:52:13.032726 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032702 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.032812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032717 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.032812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032746 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.032812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032778 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.032812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032801 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.032991 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032822 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.032991 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032822 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.032991 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.032848 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033024 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033096 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.033180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033150 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033153 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033194 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.033286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033258 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.033467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033306 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033442 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033451 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033471 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033540 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033574 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033600 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033620 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.033683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033674 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6jt\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.034018 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033711 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk2w\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.034018 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.033760 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.034237 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.034206 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.036406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.036354 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.036686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.036662 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.071840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.071814 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.072305 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.072285 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk2w\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.134022 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.133986 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:13.135050 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135014 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135203 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135127 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135238 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6jt\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135288 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135449 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135564 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135510 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135564 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135535 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135635 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.135725 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.135690 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.136193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.136165 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.136610 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.136584 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.136872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.136849 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.137286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.137252 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.137631 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.137544 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.140354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.140313 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.141960 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.141916 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.176932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.176868 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.177829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.177781 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6jt\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.232385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.231555 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:13.494463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.494436 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:52:13.496729 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:52:13.496702 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bb1deb_41a4_450c_a1c8_83ebd000e1ed.slice/crio-31173ee40161421923efc5b9e52922f841c38d32cc619ce20ca342aa7b876b2b WatchSource:0}: Error finding container 31173ee40161421923efc5b9e52922f841c38d32cc619ce20ca342aa7b876b2b: Status 404 returned error can't find the container with id 31173ee40161421923efc5b9e52922f841c38d32cc619ce20ca342aa7b876b2b Oct 30 19:52:13.499599 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.499568 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.499711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.499645 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.499711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.499688 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.575466 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:52:13.575432 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7239b6c0_a0b9_4040_b25d_ada66fd09937.slice/crio-ea602f5fef96560259803e6810cfd8281fefde11a632f0581836142bf448a466 WatchSource:0}: Error finding container ea602f5fef96560259803e6810cfd8281fefde11a632f0581836142bf448a466: Status 404 returned error can't find the container with id ea602f5fef96560259803e6810cfd8281fefde11a632f0581836142bf448a466 Oct 30 19:52:13.578336 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.578300 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.578449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.578405 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.578525 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.578487 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:13.579884 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.579859 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:52:13.657458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.657004 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" event={"ID":"5de6513d-f9e2-4166-9390-7e8ac17ef3f5","Type":"ContainerStarted","Data":"21274d707dc533ba4378c8ba966edf2cf526578e3534afb24dd9bcb3d1a2974c"} Oct 30 19:52:13.657458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.657070 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" event={"ID":"5de6513d-f9e2-4166-9390-7e8ac17ef3f5","Type":"ContainerStarted","Data":"b519a9fb4e19ea38e91eddcf4db01ee5076c3b9885522acaf460fe286eaba9f2"} Oct 30 19:52:13.662068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.661181 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" event={"ID":"58fe2ac9-4d7f-405c-ac64-236c2166abfe","Type":"ContainerStarted","Data":"1eaf5a70dc3c17288925a9ff75fdea88e974f9744b309920ba4bd212830d340e"} Oct 30 19:52:13.662068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.661242 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" event={"ID":"58fe2ac9-4d7f-405c-ac64-236c2166abfe","Type":"ContainerStarted","Data":"e3ba6061a3ec40b1e8c9f91074249ed7e316acc527dc35b3755c3f4f437d8ef1"} Oct 30 19:52:13.685383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.681309 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" event={"ID":"7239b6c0-a0b9-4040-b25d-ada66fd09937","Type":"ContainerStarted","Data":"ea602f5fef96560259803e6810cfd8281fefde11a632f0581836142bf448a466"} Oct 30 19:52:13.725837 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.725800 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" event={"ID":"59bb1deb-41a4-450c-a1c8-83ebd000e1ed","Type":"ContainerStarted","Data":"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0"} Oct 30 19:52:13.727058 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.727028 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" event={"ID":"59bb1deb-41a4-450c-a1c8-83ebd000e1ed","Type":"ContainerStarted","Data":"31173ee40161421923efc5b9e52922f841c38d32cc619ce20ca342aa7b876b2b"} Oct 30 19:52:13.729479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.728650 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podStartSLOduration=1.728632159 podStartE2EDuration="1.728632159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.723673269 +0000 UTC m=+1001.173166005" watchObservedRunningTime="2026-10-30 19:52:13.728632159 +0000 UTC m=+1001.178124885" Oct 30 19:52:13.830304 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.830166 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podStartSLOduration=1.830143982 podStartE2EDuration="1.830143982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.772080301 +0000 UTC m=+1001.221573034" watchObservedRunningTime="2026-10-30 19:52:13.830143982 +0000 UTC m=+1001.279636709" Oct 30 19:52:13.837254 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:13.837177 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podStartSLOduration=1.837163216 podStartE2EDuration="1.837163216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.81484477 +0000 UTC m=+1001.264337508" watchObservedRunningTime="2026-10-30 19:52:13.837163216 +0000 UTC m=+1001.286655942" Oct 30 19:52:14.136996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.136920 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:14.137275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.137258 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.137409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.137384 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:14.597421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.597148 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:14.597421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.597328 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.597421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.597392 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:14.626424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.624456 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.626424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.624513 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:14.626424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.624723 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:14.772388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.771680 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" event={"ID":"7239b6c0-a0b9-4040-b25d-ada66fd09937","Type":"ContainerStarted","Data":"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53"} Oct 30 19:52:14.856452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:14.854620 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podStartSLOduration=2.854599182 podStartE2EDuration="2.854599182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:14.829706389 +0000 UTC m=+1002.279199116" watchObservedRunningTime="2026-10-30 19:52:14.854599182 +0000 UTC m=+1002.304091910" Oct 30 19:52:15.140450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.139792 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.140450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.139857 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:15.240793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.240757 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:15.262985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.262943 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.263273 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.263242 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:15.739409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.733738 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.739409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.733798 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:15.743387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.740089 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.743387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:15.740156 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:16.066773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.065476 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:16.135789 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.134828 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:16.135789 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.134894 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:16.156475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.155746 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:16.335450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.310965 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:16.587449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.531555 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:16.587449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.531684 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:16.597391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.589625 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.056177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.627039 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:17.056177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.670592 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.056177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.047835 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:17.056177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.039625 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.056177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.792957 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:17.102388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:16.670693 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.102720 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.102677 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:17.208501 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.208436 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" podUID="62b90662-f37f-4ba0-a9fc-9b37bcdfe838" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.217393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.213077 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:52:17.230390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.228434 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:17.230390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.229580 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:52:17.258811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.256073 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.364871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.364761 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.365506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.365477 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.365692 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.365669 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:17.384390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.371886 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:17.447386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.407694 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.447386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.407755 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:17.513396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.484670 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:52:17.514132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.514106 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:52:17.514313 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.514301 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:52:17.606570 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.606536 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.606868 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.606826 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:17.607165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.607145 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.607335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:17.607313 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:18.061029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.060986 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:18.071483 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.071437 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:18.084300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.071555 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:18.092672 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.075819 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:52:18.157708 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.098892 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:52:18.157708 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.148518 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:52:18.240086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.240030 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:18.439503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.241434 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:52:18.491390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.241497 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:18.491390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.241601 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:52:18.491390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.470385 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:52:18.491390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.470432 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:52:18.491390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.253733 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:52:18.505388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.265616 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:18.515427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.412583 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:18.577180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.346203 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.577180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.566738 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:18.577180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.411440 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:18.577180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.347850 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.577180 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.567068 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:18.644389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.621531 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.644389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.621590 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:18.644389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.621671 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.644389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:18.621693 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:19.079856 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.079805 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:19.081253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.081219 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:19.097501 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.097462 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:19.139593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.139554 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.143586 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.143549 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:19.266385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.254675 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:52:19.278389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.277036 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.278389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.277097 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:19.278389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.277965 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:52:19.278718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.278646 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:52:19.302920 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.302871 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" podUID="1bac1e18-7114-4164-a803-75f43023a005" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:19.459731 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.459100 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:19.526507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.525323 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:19.526507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.525825 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:19.711904 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.711803 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.784450 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.712548 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.712494 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.783642 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.784881 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:19.795387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:19.713760 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.083521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.083472 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.110389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.108430 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:20.167390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.152511 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.167390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.152593 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:20.244389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.244022 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.244389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.244117 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:20.301814 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.301751 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.462771 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.462621 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.534869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.534827 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.597839 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.597807 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.603552 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.598070 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:20.624673 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.612904 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.624933 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.624904 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:20.625056 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.613110 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:20.649474 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.631100 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:52:20.716142 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.698500 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:52:20.788411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:20.788097 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:21.116415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.085546 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:21.116415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.097969 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:52:21.116415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.114227 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:52:21.118424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.118384 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:21.137888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.137856 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.142793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.142632 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:21.145123 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.141962 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:52:21.146288 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.146271 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:52:21.234410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.234246 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:52:21.235606 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.234762 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.235606 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.234817 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:21.238996 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:52:21.238946 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:52:21.595866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.595811 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.596086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.595880 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:52:21.605547 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.605509 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.605753 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:21.605587 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:52:22.135112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.135072 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:22.137193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.137138 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:52:22.232480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.232443 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:22.236669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.236622 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 19:52:22.601310 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.601278 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:22.601829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.601813 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:22.602688 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.602668 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:52:22.610205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.610178 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:22.610521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.610498 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:22.611291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:22.611271 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:52:23.142137 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.142105 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:23.142808 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.142752 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:23.143643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.143622 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:52:23.297601 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.297568 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:23.298793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.298768 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:23.305442 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:23.300743 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:52:26.698313 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:26.698277 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:52:29.103400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.102646 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:29.303534 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.303485 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.304954 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.303806 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.497392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.493179 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.513410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.504463 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:29.535275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.535234 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.574735 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.574692 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.683451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.679572 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.701667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.701612 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.793601 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:29.793494 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:30.093590 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.093464 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.110536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.110479 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.111620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.111595 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:52:30.321441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.309226 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.321441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.309669 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.321441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.310301 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:52:30.321441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.310358 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:52:30.517452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.498702 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.578412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.518462 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:52:30.578412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.519110 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.578412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.519541 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:52:30.578412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.535870 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:30.578412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.536283 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:52:30.681515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.681462 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.707310 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.707277 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:52:30.799355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:30.795551 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:31.160469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.149338 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:31.194439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.161555 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:31.194439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.184493 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:52:31.194439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.184571 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:52:31.398494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.398446 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.400561 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.400525 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.400797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.400772 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:31.630093 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.630036 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:52:31.708391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.630393 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:31.727429 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.682760 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.771547 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.771499 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.879402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.875752 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.879402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.875933 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:31.968493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:31.951592 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:52:32.288356 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.288307 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:32.567006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.288970 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.691391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.608028 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.755515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.608243 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:32.797996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.682602 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.798265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.702402 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:52:32.799340 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.702444 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:32.799894 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.741744 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:32.887924 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.887817 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.966206 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:32.966162 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.591391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.579597 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.802390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.798470 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:33.817895 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.805004 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.822392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.805894 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:33.830393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.805926 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:33.904260 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.805951 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.956419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.805973 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.956419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:33.939425 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:34.006830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.006762 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.593865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.593813 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.820435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.818475 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:34.820435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.818961 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:34.828411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.823170 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.913564 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.913454 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.978391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.958514 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.979090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.968505 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:34.979618 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:34.979586 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.008352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.007522 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.598919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.595561 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 19:52:35.600048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.599576 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:35.827393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.827283 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.847612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.827740 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:35.847612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.828357 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:35.922450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.922186 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:35.987392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.986566 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.987392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.987051 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:52:35.987654 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:35.987402 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:36.014629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.014210 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:52:36.471741 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.440515 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:52:36.471741 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.440591 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:36.661383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.650734 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:36.903578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.903463 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:52:36.919389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.905404 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:36.944422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.921269 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:52:36.945464 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.924188 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:36.945685 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:36.926323 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:37.027939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.027882 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:37.080625 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.080581 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:52:37.108382 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.100713 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:37.116550 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.111604 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:37.131253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.130979 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:52:37.131696 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.131680 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:52:37.131854 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.131843 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:52:37.132038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.132027 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:52:37.132186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.132176 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:52:37.133879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.133858 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:52:37.135107 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.134827 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:52:37.135477 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.135456 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:52:37.147911 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:37.147877 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:52:43.431736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.431675 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:52:43.438289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.438251 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.470333 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.470280 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:52:43.485918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.485872 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.485918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.485924 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486232 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.485954 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486232 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486135 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486217 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486382 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvjb\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486438 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486546 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.486699 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.486684 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588001 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.587962 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvjb\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588216 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588018 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588216 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588104 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588237 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588322 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588349 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588427 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588390 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588441 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588483 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588593 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588552 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588726 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588655 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588805 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588781 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.588967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588941 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.589038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.588989 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.590767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.590743 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.590987 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.590967 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.610038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.609996 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.610377 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.610346 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvjb\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:43.758426 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:43.758354 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:44.083905 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.083859 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:52:44.086141 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:52:44.086113 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08adf74_3256_4c8e_8de3_d34a2b31be5c.slice/crio-ac9438dc44aa5f25a7c49b9b5fdfca207bfa65c26e993f8fc935930ffb683260 WatchSource:0}: Error finding container ac9438dc44aa5f25a7c49b9b5fdfca207bfa65c26e993f8fc935930ffb683260: Status 404 returned error can't find the container with id ac9438dc44aa5f25a7c49b9b5fdfca207bfa65c26e993f8fc935930ffb683260 Oct 30 19:52:44.088820 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.088783 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:44.088929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.088865 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:44.088929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.088907 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:44.443898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.443794 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" event={"ID":"f08adf74-3256-4c8e-8de3-d34a2b31be5c","Type":"ContainerStarted","Data":"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218"} Oct 30 19:52:44.443898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.443848 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" event={"ID":"f08adf74-3256-4c8e-8de3-d34a2b31be5c","Type":"ContainerStarted","Data":"ac9438dc44aa5f25a7c49b9b5fdfca207bfa65c26e993f8fc935930ffb683260"} Oct 30 19:52:44.497095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.497032 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podStartSLOduration=1.497013873 podStartE2EDuration="1.497013873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:44.492990588 +0000 UTC m=+1031.942483313" watchObservedRunningTime="2026-10-30 19:52:44.497013873 +0000 UTC m=+1031.946506597" Oct 30 19:52:44.759489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:44.759454 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:45.760246 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:45.760211 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:45.760808 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:45.760269 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:46.759445 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:46.759403 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:46.759628 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:46.759479 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:47.758968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:47.758935 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:47.759466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:47.759000 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:48.759640 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:48.759600 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:48.760192 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:48.759668 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:49.760961 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:49.760919 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:49.762636 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:49.762120 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:50.758982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:50.758945 5991 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:52:50.759157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:50.759034 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:52:50.870923 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:52:50.870879 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:52:51.768027 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:51.767993 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:51.768431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:51.768409 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:52:51.769266 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:52:51.769246 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:53:00.127235 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.127201 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:53:00.132051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.132025 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.156089 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.156056 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:53:00.185095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.185058 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9b58e9f2\"" Oct 30 19:53:00.185299 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.185153 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9b58e9f2-dockercfg-7p6q6\"" Oct 30 19:53:00.245834 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.245797 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.246014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.245839 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5djg\" (UniqueName: \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.347663 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.347621 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.347878 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.347678 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5djg\" (UniqueName: \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.348506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.348480 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.365557 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.365524 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5djg\" (UniqueName: \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\") pod \"payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.453618 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.453480 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:00.850681 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:00.850650 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:53:00.852596 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:53:00.852556 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbc26d4_9dd7_4fd3_a539_fe36a03b9766.slice/crio-63e3b1e32d195d9263e4db31a782d359c2fe8f48dc9a08884f2f1c2fb8f61194 WatchSource:0}: Error finding container 63e3b1e32d195d9263e4db31a782d359c2fe8f48dc9a08884f2f1c2fb8f61194: Status 404 returned error can't find the container with id 63e3b1e32d195d9263e4db31a782d359c2fe8f48dc9a08884f2f1c2fb8f61194 Oct 30 19:53:01.631451 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:01.631409 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" event={"ID":"acbc26d4-9dd7-4fd3-a539-fe36a03b9766","Type":"ContainerStarted","Data":"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb"} Oct 30 19:53:01.631871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:01.631509 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:01.631871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:01.631534 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" event={"ID":"acbc26d4-9dd7-4fd3-a539-fe36a03b9766","Type":"ContainerStarted","Data":"63e3b1e32d195d9263e4db31a782d359c2fe8f48dc9a08884f2f1c2fb8f61194"} Oct 30 19:53:01.669065 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:01.668999 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" podStartSLOduration=1.668984312 podStartE2EDuration="1.668984312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:01.668579795 +0000 UTC m=+1049.118072521" watchObservedRunningTime="2026-10-30 19:53:01.668984312 +0000 UTC m=+1049.118477064" Oct 30 19:53:04.715920 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.715880 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:53:04.720914 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.720886 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.742128 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.742088 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:53:04.748450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.748415 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ffc83e66-serving-cert\"" Oct 30 19:53:04.830417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.830326 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.830639 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.830434 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdpb\" (UniqueName: \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.830705 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.830670 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.830763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.830726 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.932978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.932935 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.932978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.932986 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.933265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.933172 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.933265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.933191 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdpb\" (UniqueName: \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.933955 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.933923 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.936725 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.936680 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:04.937197 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:04.937172 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:05.103891 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:05.095859 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdpb\" (UniqueName: \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\") pod \"maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:05.344386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:05.340792 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:06.770382 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.762484 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.793616 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.793544 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.827980 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.827469 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.837392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.832725 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:06.927445 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.922397 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.973388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.971485 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.973388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:06.971940 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:07.748392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.744001 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.789404 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.782505 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:53:07.824396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.822462 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.824396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.822964 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:53:07.825302 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.825262 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.835491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.825653 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:53:07.835491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.829836 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:07.865551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.855557 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:53:07.903391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.866188 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.903391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.867290 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:53:07.998397 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.994544 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.009392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:07.998680 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.065393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:08.033061 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:08.065393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:08.033900 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:53:08.065393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:08.033964 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:53:08.260669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:08.260612 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:53:09.058393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.001518 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.058393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.016907 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:53:09.170655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.080990 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.117136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.117079 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.191525 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.191461 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.294392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.086013 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.305535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.086069 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" podUID="1bac1e18-7114-4164-a803-75f43023a005" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.305822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.124109 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.306919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.217861 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:10.333541 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.344506 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" podUID="62b90662-f37f-4ba0-a9fc-9b37bcdfe838" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.333865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.356110 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.441683 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.414093 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.362668 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.085549 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.804484 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.195413 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.288903 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:10.379014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.363384 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.389587 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.389515 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bf7cf" podUID="47a539a2-c29a-47c4-93bd-ebc74ad85856" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.414584 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:09.085607 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.453389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.417036 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" podUID="45d0120c-b448-490e-ae2b-040abdd254c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 19:53:10.700277 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.424550 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-6vn57" podUID="3d3a5c43-a952-42f9-9e67-3a629aca6b72" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.38:50052\" within 1s: context deadline exceeded" Oct 30 19:53:10.700684 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.700653 5991 patch_prober.go:29] interesting pod/thanos-querier-747576895b-zqwt6 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:10.700866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.700837 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-747576895b-zqwt6" podUID="e7977864-d9ee-43d9-9c44-6611a31dba60" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.918880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.918834 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" podUID="1bac1e18-7114-4164-a803-75f43023a005" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.938849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:10.920455 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.151656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.151478 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.321394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.320691 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw" podUID="ab0d2e72-c6c8-482c-b69f-9fd413e01569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:53:11.321394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.321021 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.321394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.321155 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:11.321394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.321300 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.341489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.341438 5991 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:11.349514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.349466 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.385486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.382103 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:11.386078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.382140 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq" podUID="62b90662-f37f-4ba0-a9fc-9b37bcdfe838" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.386531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.382162 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:53:11.411508 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.382181 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:53:11.438395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.437987 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.486392 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:53:11.452499 5991 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.362s" Oct 30 19:53:11.500597 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.500551 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:53:11.500951 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.500932 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:53:11.509407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.481721 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-td6ls" podUID="789b903b-be64-4bdd-bf2c-20dd94fbce79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.685453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.659660 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" podUID="45d0120c-b448-490e-ae2b-040abdd254c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.846383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.817884 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 30 19:53:11.897391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:11.894512 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.194408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.179529 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:53:12.223568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.223527 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.339957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.339915 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:12.340623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.340588 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:12.341076 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.341050 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.433006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.409400 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.433006 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.411144 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:12.492448 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.471475 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:53:12.589319 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.589268 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:53:12.606743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.606700 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 19:53:12.670566 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:12.664698 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.828069 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:53:12.827960 5991 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.327s" Oct 30 19:53:13.207447 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.207394 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.280407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.248757 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.381391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.348471 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.381391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.356664 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:13.381391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.356997 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:13.426391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.426304 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:13.456208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.445879 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.489450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.484597 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.489450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.486730 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:13.489450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.486793 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.600575 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.600523 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.639825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.639789 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:53:13.640118 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.640091 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" event={"ID":"f9a357a2-1963-40c9-9867-0d1a769a4482","Type":"ContainerStarted","Data":"6e463f5c986fd054eba310ea3296d903d0ba11217553d4a8630a42e65ac3f332"} Oct 30 19:53:13.640294 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.640278 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:53:13.640462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.640447 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:53:13.640630 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.640618 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ngh9x" Oct 30 19:53:13.640768 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:13.640757 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:53:14.135316 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.135274 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" event={"ID":"f9a357a2-1963-40c9-9867-0d1a769a4482","Type":"ContainerStarted","Data":"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240"} Oct 30 19:53:14.140405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.140352 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:14.221466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.211823 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.224181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.224098 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" podStartSLOduration=8.803330008 podStartE2EDuration="10.224070527s" totalImagesPullingTime="1.420740519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:14.212502084 +0000 UTC m=+1061.661994810" watchObservedRunningTime="2026-10-30 19:53:14.224070527 +0000 UTC m=+1061.673563254" Oct 30 19:53:14.225732 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.225469 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-j8486" podUID="1bac1e18-7114-4164-a803-75f43023a005" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.273041 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.259523 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:53:14.301917 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.299923 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:53:14.358355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.357110 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:14.359047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.357051 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.544098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.544062 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:53:14.548424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.548380 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:14.550196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.550164 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:53:14.595400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.592169 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:53:14.636773 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.636733 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:53:14.637422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.637387 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:53:14.637558 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.637471 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:53:14.645455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.645423 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:53:14.721389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.717945 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:53:14.721389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.718009 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:53:14.721389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:14.718047 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:53:19.652928 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.652895 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:53:19.657528 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.657507 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.670491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.670463 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dc2ad28e\"" Oct 30 19:53:19.685126 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.685088 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:53:19.696813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.696773 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:53:19.703882 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.703794 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.719154 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.719127 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dc2ad28e-dockercfg-z6l75\"" Oct 30 19:53:19.720994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.720959 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:53:19.746740 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.746707 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.746958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.746937 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.747031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.746987 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxtf\" (UniqueName: \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.747221 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.747203 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj6w\" (UniqueName: \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.884389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.855781 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.884731 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.884701 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxtf\" (UniqueName: \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.885125 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.885099 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj6w\" (UniqueName: \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.885337 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.885321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.887810 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.882323 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.887998 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.887305 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.941436 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.941211 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxtf\" (UniqueName: \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\") pod \"payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:19.962385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.961208 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj6w\" (UniqueName: \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\") pod \"payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:19.983387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:19.982636 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:20.026008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:20.025972 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:20.314256 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:20.314226 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:53:20.316441 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:53:20.316354 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4e089c_f524_46b3_a980_98946ec440d6.slice/crio-8095c5b8dbd9f41564ec9fef75857c8c7f784211f883e90096518e75e7490e38 WatchSource:0}: Error finding container 8095c5b8dbd9f41564ec9fef75857c8c7f784211f883e90096518e75e7490e38: Status 404 returned error can't find the container with id 8095c5b8dbd9f41564ec9fef75857c8c7f784211f883e90096518e75e7490e38 Oct 30 19:53:20.356341 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:20.356314 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:53:20.358730 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:53:20.358704 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f0cac2_1e70_4c2e_8543_ecfa093d921c.slice/crio-a615acde3219c8ef017013ede57f3edbf5c7173150f644478ee99f0d9ae6e1f6 WatchSource:0}: Error finding container a615acde3219c8ef017013ede57f3edbf5c7173150f644478ee99f0d9ae6e1f6: Status 404 returned error can't find the container with id a615acde3219c8ef017013ede57f3edbf5c7173150f644478ee99f0d9ae6e1f6 Oct 30 19:53:20.872566 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:53:20.872526 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:53:21.165195 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.165113 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:53:21.228634 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.228588 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" event={"ID":"42f0cac2-1e70-4c2e-8543-ecfa093d921c","Type":"ContainerStarted","Data":"236aa062a7cfea3afd4f281d268532fdbe7432363ef7ad700b39021bc7d20d01"} Oct 30 19:53:21.228830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.228671 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" event={"ID":"42f0cac2-1e70-4c2e-8543-ecfa093d921c","Type":"ContainerStarted","Data":"a615acde3219c8ef017013ede57f3edbf5c7173150f644478ee99f0d9ae6e1f6"} Oct 30 19:53:21.229651 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.229622 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:21.232074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.230921 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" event={"ID":"ac4e089c-f524-46b3-a980-98946ec440d6","Type":"ContainerStarted","Data":"062c26f8c411d425ee6acd0a9623908b1441b70b837cf28c6a46a33ae19a8149"} Oct 30 19:53:21.232074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.230983 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" event={"ID":"ac4e089c-f524-46b3-a980-98946ec440d6","Type":"ContainerStarted","Data":"8095c5b8dbd9f41564ec9fef75857c8c7f784211f883e90096518e75e7490e38"} Oct 30 19:53:21.232074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.231796 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:21.273905 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.273827 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" podStartSLOduration=2.273807744 podStartE2EDuration="2.273807744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:21.273077294 +0000 UTC m=+1068.722570018" watchObservedRunningTime="2026-10-30 19:53:21.273807744 +0000 UTC m=+1068.723300470" Oct 30 19:53:21.310834 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:21.310776 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" podStartSLOduration=2.310758158 podStartE2EDuration="2.310758158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:21.307098383 +0000 UTC m=+1068.756591111" watchObservedRunningTime="2026-10-30 19:53:21.310758158 +0000 UTC m=+1068.760250882" Oct 30 19:53:29.299627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.299590 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:53:29.306703 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.306667 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.329091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.329061 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c9c6d81d-openshift-default-dockercfg-7f74q\"" Oct 30 19:53:29.345350 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.345317 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:53:29.358806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.358536 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.358806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.358589 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.358806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.358759 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.358806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.358792 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.359176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.358895 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.359176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.359096 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w68d\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.359176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.359141 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.359423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.359197 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.359423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.359255 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461088 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461049 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w68d\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461322 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461096 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461322 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461133 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461525 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461400 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461612 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461589 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461660 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461613 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461660 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461636 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461708 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461808 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461790 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461796 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461816 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461848 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.461995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.461880 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.462036 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.462002 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.464111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.464088 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.464489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.464469 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.480511 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.480413 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w68d\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.490184 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.490152 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\") pod \"e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.649297 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.649207 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:29.987414 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.987378 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:53:29.988729 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:53:29.988701 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63478f6_c484_428b_bf9c_a2ada4814ac7.slice/crio-3c7c74e1bde6313b9f5acacc0a07c0f010fc3aa20e6819d27d3e29a6d13ed373 WatchSource:0}: Error finding container 3c7c74e1bde6313b9f5acacc0a07c0f010fc3aa20e6819d27d3e29a6d13ed373: Status 404 returned error can't find the container with id 3c7c74e1bde6313b9f5acacc0a07c0f010fc3aa20e6819d27d3e29a6d13ed373 Oct 30 19:53:29.991643 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.991613 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:29.991721 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.991677 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:29.991721 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:29.991706 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:30.333508 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:30.333346 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" event={"ID":"e63478f6-c484-428b-bf9c-a2ada4814ac7","Type":"ContainerStarted","Data":"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6"} Oct 30 19:53:30.334102 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:30.334064 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" event={"ID":"e63478f6-c484-428b-bf9c-a2ada4814ac7","Type":"ContainerStarted","Data":"3c7c74e1bde6313b9f5acacc0a07c0f010fc3aa20e6819d27d3e29a6d13ed373"} Oct 30 19:53:30.649880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:30.649798 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:31.656536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:31.656501 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:31.707061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:31.707003 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" podStartSLOduration=2.706988405 podStartE2EDuration="2.706988405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:30.391199476 +0000 UTC m=+1077.840692201" watchObservedRunningTime="2026-10-30 19:53:31.706988405 +0000 UTC m=+1079.156481130" Oct 30 19:53:32.241879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:32.241848 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:53:32.352838 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:32.352798 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:32.353992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:32.353963 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:53:33.249193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:33.249161 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:53:34.982330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:34.982290 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:53:34.988245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:34.988218 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.014607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.014527 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dc2ad28e-serving-cert\"" Oct 30 19:53:35.024289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.024256 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:53:35.059345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.059311 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.059550 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.059509 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.059683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.059666 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.059780 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.059761 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9ms\" (UniqueName: \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.161535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.161491 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9ms\" (UniqueName: \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.161746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.161691 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.161823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.161765 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.161881 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.161862 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.162486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.162460 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.164466 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.164443 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.164599 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.164530 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.198316 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.198282 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9ms\" (UniqueName: \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\") pod \"maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.307733 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.306595 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:35.640712 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:35.640684 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:53:35.641417 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:53:35.641385 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff30f5e_142b_4cc6_a6f9_bcfe687cae4b.slice/crio-d7824d454d62ea3599a611d2a64c677ba815a62e69547b1a9e7e0cbd5f3dca4e WatchSource:0}: Error finding container d7824d454d62ea3599a611d2a64c677ba815a62e69547b1a9e7e0cbd5f3dca4e: Status 404 returned error can't find the container with id d7824d454d62ea3599a611d2a64c677ba815a62e69547b1a9e7e0cbd5f3dca4e Oct 30 19:53:36.390736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:36.390645 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" event={"ID":"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b","Type":"ContainerStarted","Data":"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997"} Oct 30 19:53:36.390736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:36.390705 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" event={"ID":"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b","Type":"ContainerStarted","Data":"d7824d454d62ea3599a611d2a64c677ba815a62e69547b1a9e7e0cbd5f3dca4e"} Oct 30 19:53:36.390736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:36.390733 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:36.472958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:36.466726 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" podStartSLOduration=2.008511636 podStartE2EDuration="2.466705336s" totalImagesPullingTime="458.1937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:36.431473677 +0000 UTC m=+1083.880966404" watchObservedRunningTime="2026-10-30 19:53:36.466705336 +0000 UTC m=+1083.916198062" Oct 30 19:53:42.407080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:42.407050 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:53:43.960441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:43.960147 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:53:46.303003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.302955 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.314186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.313453 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.314532 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.314499 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.355941 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.355531 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.418515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.418462 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.438515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.438462 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.473507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.473457 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.636691 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.636577 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.973449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:46.963971 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 19:53:47.337449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.211066 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.415620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.415570 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:53:47.417136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.416526 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:47.442497 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.442446 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:47.485690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.447793 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:47.485690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.471402 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:53:47.485690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.471565 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:53:47.505449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.496576 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:53:47.505449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.496700 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:53:47.570390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.513103 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.570908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.570877 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:53:47.692457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.658639 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.892759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.700637 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.133396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.700662 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.205450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:47.894603 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:53:48.213565 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.213528 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:53:48.402533 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.402450 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:53:48.448417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.281452 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.767786 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.448675 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.824964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.400911 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.824964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.452973 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:48.839410 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.473017 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:49.005388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.473067 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:49.038446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:48.805888 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:53:49.222701 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.222626 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:53:49.543750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.543705 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:53:49.552531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.552501 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583111 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583201 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583238 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583278 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583309 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.583337 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 19:53:49.584633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.584105 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 19:53:49.632602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.619831 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 19:53:49.751200 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:49.733786 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.872044 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:53:50.871996 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:53:59.893160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:59.893123 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:53:59.893715 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:53:59.893334 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" podUID="f9a357a2-1963-40c9-9867-0d1a769a4482" containerName="maas-api" containerID="cri-o://d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240" gracePeriod=30 Oct 30 19:54:00.208542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.208516 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:54:00.318407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.318354 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\" (UniqueName: \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\") pod \"f9a357a2-1963-40c9-9867-0d1a769a4482\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " Oct 30 19:54:00.318691 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.318669 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\") pod \"f9a357a2-1963-40c9-9867-0d1a769a4482\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " Oct 30 19:54:00.319043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.319023 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\") pod \"f9a357a2-1963-40c9-9867-0d1a769a4482\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " Oct 30 19:54:00.320357 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.319973 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca" pod "f9a357a2-1963-40c9-9867-0d1a769a4482" (UID: "f9a357a2-1963-40c9-9867-0d1a769a4482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:00.320942 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.320755 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\") pod \"f9a357a2-1963-40c9-9867-0d1a769a4482\" (UID: \"f9a357a2-1963-40c9-9867-0d1a769a4482\") " Oct 30 19:54:00.322621 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.322603 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a357a2-1963-40c9-9867-0d1a769a4482-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:00.340638 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.334323 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb" pod "f9a357a2-1963-40c9-9867-0d1a769a4482" (UID: "f9a357a2-1963-40c9-9867-0d1a769a4482"). InnerVolumeSpecName "kube-api-access-pmdpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:00.356736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.356665 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls" pod "f9a357a2-1963-40c9-9867-0d1a769a4482" (UID: "f9a357a2-1963-40c9-9867-0d1a769a4482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:00.359744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.359696 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs" pod "f9a357a2-1963-40c9-9867-0d1a769a4482" (UID: "f9a357a2-1963-40c9-9867-0d1a769a4482"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:00.424345 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.424237 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:00.424631 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.424581 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a357a2-1963-40c9-9867-0d1a769a4482-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:00.424631 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.424608 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdpb\" (UniqueName: \"kubernetes.io/projected/f9a357a2-1963-40c9-9867-0d1a769a4482-kube-api-access-pmdpb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:00.461994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.461959 5991 generic.go:356] "Generic (PLEG): container finished" podID="f9a357a2-1963-40c9-9867-0d1a769a4482" containerID="d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240" exitCode=0 Oct 30 19:54:00.462191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.462009 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" Oct 30 19:54:00.462191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.462037 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" event={"ID":"f9a357a2-1963-40c9-9867-0d1a769a4482","Type":"ContainerDied","Data":"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240"} Oct 30 19:54:00.462191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.462087 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658" event={"ID":"f9a357a2-1963-40c9-9867-0d1a769a4482","Type":"ContainerDied","Data":"6e463f5c986fd054eba310ea3296d903d0ba11217553d4a8630a42e65ac3f332"} Oct 30 19:54:00.462191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.462109 5991 scope.go:122] "RemoveContainer" containerID="d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240" Oct 30 19:54:00.475137 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.475110 5991 scope.go:122] "RemoveContainer" containerID="d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240" Oct 30 19:54:00.475515 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:00.475486 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240\": container with ID starting with d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240 not found: ID does not exist" containerID="d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240" Oct 30 19:54:00.475605 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.475527 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240"} err="failed to get container status \"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240\": rpc error: code = NotFound desc = could not find container \"d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240\": container with ID starting with d12053fa0d16fd22c36178f2ab776d55f15b0dcc53cba788101189d8a0d9a240 not found: ID does not exist" Oct 30 19:54:00.509690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.509655 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:54:00.519903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.519867 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffc83e66-6cbb6f7878-8f658"] Oct 30 19:54:00.682878 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:00.682798 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a357a2-1963-40c9-9867-0d1a769a4482" path="/var/lib/kubelet/pods/f9a357a2-1963-40c9-9867-0d1a769a4482/volumes" Oct 30 19:54:01.015901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:01.015871 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:54:06.016977 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.016933 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" podUID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" containerName="payload-pre-processing" containerID="cri-o://afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb" gracePeriod=25 Oct 30 19:54:06.320287 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.320252 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:54:06.416388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.405502 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\" (UniqueName: \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\") pod \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " Oct 30 19:54:06.416388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.405599 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\") pod \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\" (UID: \"acbc26d4-9dd7-4fd3-a539-fe36a03b9766\") " Oct 30 19:54:06.417772 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.416981 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume" pod "acbc26d4-9dd7-4fd3-a539-fe36a03b9766" (UID: "acbc26d4-9dd7-4fd3-a539-fe36a03b9766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:06.509711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.508733 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:06.510891 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.510848 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg" pod "acbc26d4-9dd7-4fd3-a539-fe36a03b9766" (UID: "acbc26d4-9dd7-4fd3-a539-fe36a03b9766"). InnerVolumeSpecName "kube-api-access-n5djg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:06.523503 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.523414 5991 generic.go:356] "Generic (PLEG): container finished" podID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" containerID="afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb" exitCode=0 Oct 30 19:54:06.524034 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.523999 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" event={"ID":"acbc26d4-9dd7-4fd3-a539-fe36a03b9766","Type":"ContainerDied","Data":"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb"} Oct 30 19:54:06.524209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.524193 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" event={"ID":"acbc26d4-9dd7-4fd3-a539-fe36a03b9766","Type":"ContainerDied","Data":"63e3b1e32d195d9263e4db31a782d359c2fe8f48dc9a08884f2f1c2fb8f61194"} Oct 30 19:54:06.524329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.524316 5991 scope.go:122] "RemoveContainer" containerID="afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb" Oct 30 19:54:06.524673 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.524629 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt" Oct 30 19:54:06.559475 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.559454 5991 scope.go:122] "RemoveContainer" containerID="afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb" Oct 30 19:54:06.560062 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:06.560031 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb\": container with ID starting with afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb not found: ID does not exist" containerID="afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb" Oct 30 19:54:06.560170 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.560076 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb"} err="failed to get container status \"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb\": rpc error: code = NotFound desc = could not find container \"afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb\": container with ID starting with afd8c701e22db81bebe3971ea7b0fedc66031dc16a33344dde1eec95dd6f2dbb not found: ID does not exist" Oct 30 19:54:06.597413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.593385 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:54:06.598185 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.598148 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b58e9f2-7466f97c9-nbsvt"] Oct 30 19:54:06.611045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.611010 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5djg\" (UniqueName: \"kubernetes.io/projected/acbc26d4-9dd7-4fd3-a539-fe36a03b9766-kube-api-access-n5djg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:06.641748 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.641698 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:54:06.642517 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642499 5991 cpu_manager.go:414] "containerMap: removing container" podUID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" containerName="payload-pre-processing" Oct 30 19:54:06.642583 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642520 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" containerName="payload-pre-processing" Oct 30 19:54:06.642583 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642550 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f9a357a2-1963-40c9-9867-0d1a769a4482" containerName="maas-api" Oct 30 19:54:06.642583 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642555 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a357a2-1963-40c9-9867-0d1a769a4482" containerName="maas-api" Oct 30 19:54:06.642713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642682 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" containerName="payload-pre-processing" Oct 30 19:54:06.642713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.642694 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a357a2-1963-40c9-9867-0d1a769a4482" containerName="maas-api" Oct 30 19:54:06.648125 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.648093 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.662996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.662966 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-75082b11-dockercfg-z6rfp\"" Oct 30 19:54:06.663263 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.663249 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-75082b11\"" Oct 30 19:54:06.669840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.669806 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:54:06.685997 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.685961 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbc26d4-9dd7-4fd3-a539-fe36a03b9766" path="/var/lib/kubelet/pods/acbc26d4-9dd7-4fd3-a539-fe36a03b9766/volumes" Oct 30 19:54:06.712667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.712632 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.712874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.712695 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqmp\" (UniqueName: \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.715892 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.715860 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:54:06.721873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.721848 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.758210 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.758182 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:54:06.815627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.815515 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.815627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.815571 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.815880 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.815738 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqmp\" (UniqueName: \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.815940 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.815904 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9nf\" (UniqueName: \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.816544 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.816515 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.834579 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.834543 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqmp\" (UniqueName: \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\") pod \"payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:06.918318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.918273 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.918580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.918476 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9nf\" (UniqueName: \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.919183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.919157 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.937104 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.937062 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9nf\" (UniqueName: \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\") pod \"payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:06.966141 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:06.966094 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:07.061043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.060782 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:07.335544 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.335476 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:54:07.470377 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.470327 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:54:07.481512 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:54:07.481435 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa447404_f8f6_45d8_9821_6d855aba154d.slice/crio-68977c1df76d60b96aee8999addd90e0a47c98044c62777efbfdc545bb7c5c27 WatchSource:0}: Error finding container 68977c1df76d60b96aee8999addd90e0a47c98044c62777efbfdc545bb7c5c27: Status 404 returned error can't find the container with id 68977c1df76d60b96aee8999addd90e0a47c98044c62777efbfdc545bb7c5c27 Oct 30 19:54:07.536490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.536448 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" event={"ID":"3d46d994-5f56-45ec-9395-77ab919b0f5c","Type":"ContainerStarted","Data":"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f"} Oct 30 19:54:07.536707 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.536517 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" event={"ID":"3d46d994-5f56-45ec-9395-77ab919b0f5c","Type":"ContainerStarted","Data":"1ea6b5963996fc4534e2c8fe373be35c927e9386ccc50370a90e277f169d4da0"} Oct 30 19:54:07.537765 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.537692 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:07.541009 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.540981 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" event={"ID":"aa447404-f8f6-45d8-9821-6d855aba154d","Type":"ContainerStarted","Data":"68977c1df76d60b96aee8999addd90e0a47c98044c62777efbfdc545bb7c5c27"} Oct 30 19:54:07.581500 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:07.581434 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" podStartSLOduration=1.5814027899999998 podStartE2EDuration="1.58140279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:07.575971798 +0000 UTC m=+1115.025464523" watchObservedRunningTime="2026-10-30 19:54:07.58140279 +0000 UTC m=+1115.030895515" Oct 30 19:54:08.552847 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:08.552792 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" event={"ID":"aa447404-f8f6-45d8-9821-6d855aba154d","Type":"ContainerStarted","Data":"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4"} Oct 30 19:54:08.553412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:08.552931 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:08.592402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:08.592307 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" podStartSLOduration=2.592294166 podStartE2EDuration="2.592294166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:08.588201141 +0000 UTC m=+1116.037693871" watchObservedRunningTime="2026-10-30 19:54:08.592294166 +0000 UTC m=+1116.041786891" Oct 30 19:54:09.070187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.070151 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 19:54:09.075286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.075258 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.104070 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.104014 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 19:54:09.190582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190541 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.190794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190653 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.190794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190706 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.190794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190728 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.190968 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190898 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.191019 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.190995 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.191100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.191084 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.191157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.191112 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kmp\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.191230 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.191217 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.292867 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.292821 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.292887 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.292923 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.292940 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293009 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293067 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293151 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293177 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kmp\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293289 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293599 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293466 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293599 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293585 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293705 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293676 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293730 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.293833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.293811 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.295682 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.295660 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.295831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.295813 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.310953 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.310920 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.311080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.311025 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kmp\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lq4zf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.398213 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.398118 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:09.733659 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.733626 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 19:54:09.734027 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:54:09.733836 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e3272_eecd_4153_80b1_5b628e1221cf.slice/crio-7de6ab9db4a4c17f16eb8cb6341d5a1ea452660880b8e62ac2924ec2ef9c79c8 WatchSource:0}: Error finding container 7de6ab9db4a4c17f16eb8cb6341d5a1ea452660880b8e62ac2924ec2ef9c79c8: Status 404 returned error can't find the container with id 7de6ab9db4a4c17f16eb8cb6341d5a1ea452660880b8e62ac2924ec2ef9c79c8 Oct 30 19:54:09.736716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.736684 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:09.736803 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.736753 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:09.736803 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:09.736783 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:10.596884 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:10.596841 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" event={"ID":"204e3272-eecd-4153-80b1-5b628e1221cf","Type":"ContainerStarted","Data":"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617"} Oct 30 19:54:10.597096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:10.596912 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" event={"ID":"204e3272-eecd-4153-80b1-5b628e1221cf","Type":"ContainerStarted","Data":"7de6ab9db4a4c17f16eb8cb6341d5a1ea452660880b8e62ac2924ec2ef9c79c8"} Oct 30 19:54:10.644330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:10.644275 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podStartSLOduration=1.6442600760000001 podStartE2EDuration="1.644260076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:10.641253485 +0000 UTC m=+1118.090746211" watchObservedRunningTime="2026-10-30 19:54:10.644260076 +0000 UTC m=+1118.093752887" Oct 30 19:54:11.399294 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:11.399249 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:11.401320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:11.401288 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:11.401528 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:11.401352 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:12.413653 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:12.413611 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:12.414169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:12.413692 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:13.398471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:13.398438 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:13.398655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:13.398492 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:14.399078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:14.399033 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:14.399531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:14.399115 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:15.399736 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:15.399702 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:15.400117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:15.399768 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:16.400101 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:16.399744 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:16.400101 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:16.399848 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:17.399205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:17.399158 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:17.399441 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:17.399235 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:54:18.568458 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:18.568418 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:18.569334 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:18.569309 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:18.572211 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:18.572188 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 19:54:18.609170 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:18.609140 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:54:18.641145 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:18.641114 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:54:19.574830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:19.574795 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:54:19.576707 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:19.575830 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:54:20.871281 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:20.871236 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:54:21.226871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.226837 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:54:21.227138 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.227077 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" podUID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" containerName="maas-api" containerID="cri-o://83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997" gracePeriod=30 Oct 30 19:54:21.584281 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.584241 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:54:21.767806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.767712 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\") pod \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " Oct 30 19:54:21.768074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.767930 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\") pod \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " Oct 30 19:54:21.768074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.768019 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\" (UniqueName: \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\") pod \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " Oct 30 19:54:21.768261 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.768243 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\") pod \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\" (UID: \"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b\") " Oct 30 19:54:21.768750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.768717 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca" pod "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" (UID: "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:21.769289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.769263 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:21.782393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.773204 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls" pod "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" (UID: "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:21.782393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.773240 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms" pod "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" (UID: "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b"). InnerVolumeSpecName "kube-api-access-6j9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:21.782393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.777056 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs" pod "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" (UID: "0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:21.789828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.789513 5991 generic.go:356] "Generic (PLEG): container finished" podID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" containerID="83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997" exitCode=0 Oct 30 19:54:21.789828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.789560 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" event={"ID":"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b","Type":"ContainerDied","Data":"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997"} Oct 30 19:54:21.789828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.789614 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" event={"ID":"0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b","Type":"ContainerDied","Data":"d7824d454d62ea3599a611d2a64c677ba815a62e69547b1a9e7e0cbd5f3dca4e"} Oct 30 19:54:21.789828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.789643 5991 scope.go:122] "RemoveContainer" containerID="83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997" Oct 30 19:54:21.789828 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.789807 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz" Oct 30 19:54:21.853576 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.846333 5991 scope.go:122] "RemoveContainer" containerID="83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997" Oct 30 19:54:21.853576 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:21.846829 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997\": container with ID starting with 83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997 not found: ID does not exist" containerID="83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997" Oct 30 19:54:21.853576 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.846917 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997"} err="failed to get container status \"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997\": rpc error: code = NotFound desc = could not find container \"83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997\": container with ID starting with 83d737dc9ffb070b713c51ff07814185546233c1eca69be45cfcc3e350e4b997 not found: ID does not exist" Oct 30 19:54:21.864437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.862505 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:54:21.868208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.868175 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc2ad28e-6f8684b844-jf6jz"] Oct 30 19:54:21.871839 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.871809 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:21.872330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.872313 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:21.872459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:21.872442 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9ms\" (UniqueName: \"kubernetes.io/projected/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b-kube-api-access-6j9ms\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:22.681094 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:22.681048 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" path="/var/lib/kubelet/pods/0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b/volumes" Oct 30 19:54:23.610384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.610333 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" podUID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" containerName="payload-processing" containerID="cri-o://236aa062a7cfea3afd4f281d268532fdbe7432363ef7ad700b39021bc7d20d01" gracePeriod=25 Oct 30 19:54:23.642091 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.642041 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" containerName="payload-pre-processing" containerID="cri-o://062c26f8c411d425ee6acd0a9623908b1441b70b837cf28c6a46a33ae19a8149" gracePeriod=25 Oct 30 19:54:23.809406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.808346 5991 generic.go:356] "Generic (PLEG): container finished" podID="ac4e089c-f524-46b3-a980-98946ec440d6" containerID="062c26f8c411d425ee6acd0a9623908b1441b70b837cf28c6a46a33ae19a8149" exitCode=0 Oct 30 19:54:23.809406 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.808525 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" event={"ID":"ac4e089c-f524-46b3-a980-98946ec440d6","Type":"ContainerDied","Data":"062c26f8c411d425ee6acd0a9623908b1441b70b837cf28c6a46a33ae19a8149"} Oct 30 19:54:23.811136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.811006 5991 generic.go:356] "Generic (PLEG): container finished" podID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" containerID="236aa062a7cfea3afd4f281d268532fdbe7432363ef7ad700b39021bc7d20d01" exitCode=0 Oct 30 19:54:23.811136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.811074 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" event={"ID":"42f0cac2-1e70-4c2e-8543-ecfa093d921c","Type":"ContainerDied","Data":"236aa062a7cfea3afd4f281d268532fdbe7432363ef7ad700b39021bc7d20d01"} Oct 30 19:54:23.944169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.944134 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:54:23.999099 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:23.999066 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:54:24.023444 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.023412 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\") pod \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " Oct 30 19:54:24.023656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.023594 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\") pod \"ac4e089c-f524-46b3-a980-98946ec440d6\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " Oct 30 19:54:24.023735 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.023683 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\" (UniqueName: \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\") pod \"ac4e089c-f524-46b3-a980-98946ec440d6\" (UID: \"ac4e089c-f524-46b3-a980-98946ec440d6\") " Oct 30 19:54:24.023796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.023757 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\" (UniqueName: \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\") pod \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\" (UID: \"42f0cac2-1e70-4c2e-8543-ecfa093d921c\") " Oct 30 19:54:24.023859 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.023795 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume" pod "42f0cac2-1e70-4c2e-8543-ecfa093d921c" (UID: "42f0cac2-1e70-4c2e-8543-ecfa093d921c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:24.024069 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.024043 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume" pod "ac4e089c-f524-46b3-a980-98946ec440d6" (UID: "ac4e089c-f524-46b3-a980-98946ec440d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:24.024561 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.024537 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f0cac2-1e70-4c2e-8543-ecfa093d921c-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.024683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.024566 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4e089c-f524-46b3-a980-98946ec440d6-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.026584 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.026557 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf" pod "42f0cac2-1e70-4c2e-8543-ecfa093d921c" (UID: "42f0cac2-1e70-4c2e-8543-ecfa093d921c"). InnerVolumeSpecName "kube-api-access-hrxtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:24.026825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.026790 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w" pod "ac4e089c-f524-46b3-a980-98946ec440d6" (UID: "ac4e089c-f524-46b3-a980-98946ec440d6"). InnerVolumeSpecName "kube-api-access-lxj6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:24.054812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.054741 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4"] Oct 30 19:54:24.056025 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.055995 5991 cpu_manager.go:414] "containerMap: removing container" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" containerName="payload-pre-processing" Oct 30 19:54:24.056025 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056022 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" containerName="payload-pre-processing" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056077 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" containerName="maas-api" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056085 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" containerName="maas-api" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056103 5991 cpu_manager.go:414] "containerMap: removing container" podUID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" containerName="payload-processing" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056111 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" containerName="payload-processing" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056286 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" containerName="payload-pre-processing" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056313 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff30f5e-142b-4cc6-a6f9-bcfe687cae4b" containerName="maas-api" Oct 30 19:54:24.056320 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.056323 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" containerName="payload-processing" Oct 30 19:54:24.061913 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.061848 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.084648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.084614 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4"] Oct 30 19:54:24.126819 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.126770 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.126819 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.126834 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.126893 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.126964 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqck\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-kube-api-access-4pqck\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127095 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127164 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127216 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127436 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127488 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.127772 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127568 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj6w\" (UniqueName: \"kubernetes.io/projected/ac4e089c-f524-46b3-a980-98946ec440d6-kube-api-access-lxj6w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.127772 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.127584 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxtf\" (UniqueName: \"kubernetes.io/projected/42f0cac2-1e70-4c2e-8543-ecfa093d921c-kube-api-access-hrxtf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.229336 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229292 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229445 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229585 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229509 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229687 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229764 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229731 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229838 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229826 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229899 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229851 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229899 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229870 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.229899 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229894 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.230057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.229983 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqck\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-kube-api-access-4pqck\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.230057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.230022 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.230218 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.230172 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.230934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.230707 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.230934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.230700 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.232725 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.232705 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.238088 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.238050 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.248148 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.248115 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.248450 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.248356 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqck\" (UniqueName: \"kubernetes.io/projected/c9fda10e-bbc9-4010-b06d-a2c2d7427f96-kube-api-access-4pqck\") pod \"maas-default-gateway-openshift-default-75bf867b6c-psrq4\" (UID: \"c9fda10e-bbc9-4010-b06d-a2c2d7427f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.380656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.380614 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:24.497717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.497625 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:54:24.505399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.505347 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.524844 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.524796 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c53b7887-openshift-default-dockercfg-5kwrz\"" Oct 30 19:54:24.534687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.534653 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.534687 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.534704 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.534956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.534756 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.534956 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.534938 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.535071 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.534984 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.535167 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.535145 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.535250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.535202 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.535250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.535237 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.535380 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.535286 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995t7\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.539744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.539706 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:54:24.683413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.652852 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.684071 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.684027 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.684477 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.684439 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.684686 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.684655 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.684829 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.684813 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.685009 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.684977 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995t7\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.685482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.685459 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.699411 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.685626 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.692928 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.698481 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.698818 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.698968 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.701766 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.704849 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.726919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.726895 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.727709 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.727672 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.770231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.770099 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4"] Oct 30 19:54:24.815049 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.813058 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:24.815049 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.813155 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:24.815049 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.813199 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:24.826947 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.826890 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.829840 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.829812 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995t7\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\") pod \"e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:24.832238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.832215 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" Oct 30 19:54:24.832863 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.832319 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd" event={"ID":"ac4e089c-f524-46b3-a980-98946ec440d6","Type":"ContainerDied","Data":"8095c5b8dbd9f41564ec9fef75857c8c7f784211f883e90096518e75e7490e38"} Oct 30 19:54:24.832863 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.832415 5991 scope.go:122] "RemoveContainer" containerID="062c26f8c411d425ee6acd0a9623908b1441b70b837cf28c6a46a33ae19a8149" Oct 30 19:54:24.842887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.842325 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" event={"ID":"c9fda10e-bbc9-4010-b06d-a2c2d7427f96","Type":"ContainerStarted","Data":"8c601a36fe5e7872d1707e729739ed133000a0ca33f7aae5a5149dfd668fc29b"} Oct 30 19:54:24.844452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.844102 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" Oct 30 19:54:24.844452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.844174 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42" event={"ID":"42f0cac2-1e70-4c2e-8543-ecfa093d921c","Type":"ContainerDied","Data":"a615acde3219c8ef017013ede57f3edbf5c7173150f644478ee99f0d9ae6e1f6"} Oct 30 19:54:24.852655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.852628 5991 scope.go:122] "RemoveContainer" containerID="236aa062a7cfea3afd4f281d268532fdbe7432363ef7ad700b39021bc7d20d01" Oct 30 19:54:24.917514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.917445 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:54:24.925645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.924716 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc2ad28e-5877597447-vjshd"] Oct 30 19:54:24.945494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.945263 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:54:24.954816 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:24.954786 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc2ad28e-b9499896d-f4t42"] Oct 30 19:54:25.124932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.124843 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:25.456821 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.456768 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:54:25.459908 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:54:25.459876 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8e8ce5_3da1_45ba_9623_72abd65ed82e.slice/crio-18f1768a2c846a127d60b29aaa67f350bb6098103424a669f463d949d9571ecf WatchSource:0}: Error finding container 18f1768a2c846a127d60b29aaa67f350bb6098103424a669f463d949d9571ecf: Status 404 returned error can't find the container with id 18f1768a2c846a127d60b29aaa67f350bb6098103424a669f463d949d9571ecf Oct 30 19:54:25.463350 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.463309 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:25.463497 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.463427 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:25.463497 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.463469 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:25.678399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.676044 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:54:25.885067 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.885032 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerStarted","Data":"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6"} Oct 30 19:54:25.885609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.885508 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerStarted","Data":"18f1768a2c846a127d60b29aaa67f350bb6098103424a669f463d949d9571ecf"} Oct 30 19:54:25.890833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.890801 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" event={"ID":"c9fda10e-bbc9-4010-b06d-a2c2d7427f96","Type":"ContainerStarted","Data":"41e80467ba0acc3b75c479ef06c847c5ac3cc53fb78dc4ffa61237051e495867"} Oct 30 19:54:25.944623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.944552 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podStartSLOduration=1.944530276 podStartE2EDuration="1.944530276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:25.940840911 +0000 UTC m=+1133.390333679" watchObservedRunningTime="2026-10-30 19:54:25.944530276 +0000 UTC m=+1133.394023003" Oct 30 19:54:25.986385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:25.985358 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podStartSLOduration=1.9853384969999999 podStartE2EDuration="1.985338497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:25.983057583 +0000 UTC m=+1133.432550347" watchObservedRunningTime="2026-10-30 19:54:25.985338497 +0000 UTC m=+1133.434831222" Oct 30 19:54:26.125954 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.125922 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:26.127767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.127737 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:26.127920 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.127799 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:54:26.388412 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.383741 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:26.390018 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.389980 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:26.390202 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.390050 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:26.712949 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.712873 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f0cac2-1e70-4c2e-8543-ecfa093d921c" path="/var/lib/kubelet/pods/42f0cac2-1e70-4c2e-8543-ecfa093d921c/volumes" Oct 30 19:54:26.713582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:26.713554 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" path="/var/lib/kubelet/pods/ac4e089c-f524-46b3-a980-98946ec440d6/volumes" Oct 30 19:54:27.131987 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:27.131951 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:27.382446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:27.381820 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:27.382446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:27.381885 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:27.918449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:27.918332 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:27.919572 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:27.919540 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:54:28.388901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:28.388854 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:28.389473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:28.388918 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:29.404389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:29.392090 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:29.404389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:29.392151 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:30.382253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:30.382215 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:30.382609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:30.382577 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:31.395423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:31.386454 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.395423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:31.386525 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:32.394989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:32.394950 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:32.395313 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:32.395267 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:33.387648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.386482 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.387648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.386556 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 19:54:33.805400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.804474 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 19:54:33.824595 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.824561 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:33.951392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.923569 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:54:34.087781 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.928899 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:54:34.088106 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.929032 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f48d6\"" Oct 30 19:54:34.088251 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.929071 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:54:34.088355 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.953000 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.088579 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.088556 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqw7\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.088865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.088842 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.089079 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.089062 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.089233 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.089218 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.097580 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.097540 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.098001 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.097976 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.120980 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:33.953092 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 19:54:34.222403 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.222321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.222927 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.222901 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.223127 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.223110 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.223275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.223258 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.223943 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.223921 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.225516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.224994 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.225664 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.225416 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.225779 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.225528 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.226181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.226162 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.226526 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.226494 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqw7\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.230437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.228276 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.230437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.228669 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.241731 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.241700 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.246444 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.246420 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqw7\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q97d\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.440405 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.440290 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:34.818821 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.818782 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 19:54:34.820906 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:54:34.820666 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd83d175_5490_4ebd_a249_60d1c7851aba.slice/crio-17e9dc18771a5943dcbe596f5a555e9462b402aa4fbe3590a00c2ab76473ddcd WatchSource:0}: Error finding container 17e9dc18771a5943dcbe596f5a555e9462b402aa4fbe3590a00c2ab76473ddcd: Status 404 returned error can't find the container with id 17e9dc18771a5943dcbe596f5a555e9462b402aa4fbe3590a00c2ab76473ddcd Oct 30 19:54:34.976469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.976437 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:34.976995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.976974 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:34.978401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:34.978342 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" Oct 30 19:54:35.174419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:35.174345 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" event={"ID":"dd83d175-5490-4ebd-a249-60d1c7851aba","Type":"ContainerStarted","Data":"17e9dc18771a5943dcbe596f5a555e9462b402aa4fbe3590a00c2ab76473ddcd"} Oct 30 19:54:37.379549 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:37.379514 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:37.379891 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:37.379580 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:38.215833 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:38.215792 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" event={"ID":"dd83d175-5490-4ebd-a249-60d1c7851aba","Type":"ContainerStarted","Data":"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734"} Oct 30 19:54:38.216038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:38.215964 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:38.218081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:38.218045 5991 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7q97d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:54:38.218222 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:38.218110 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:54:38.280741 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:38.280674 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" podStartSLOduration=2.728784835 podStartE2EDuration="5.280652937s" totalImagesPullingTime="2.551868102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:38.264725064 +0000 UTC m=+1145.714217789" watchObservedRunningTime="2026-10-30 19:54:38.280652937 +0000 UTC m=+1145.730145663" Oct 30 19:54:39.047787 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:39.047748 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:54:39.048318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:39.047983 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" podUID="e63478f6-c484-428b-bf9c-a2ada4814ac7" containerName="istio-proxy" containerID="cri-o://f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6" gracePeriod=30 Oct 30 19:54:39.224097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:39.224065 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 19:54:40.058661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.058631 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:54:40.185898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.185810 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.185942 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186096 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:40.186160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186101 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186153 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186244 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186233 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186307 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186295 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186307 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:40.186417 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186378 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:40.186545 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186443 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186545 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186502 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.186545 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186537 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\") pod \"e63478f6-c484-428b-bf9c-a2ada4814ac7\" (UID: \"e63478f6-c484-428b-bf9c-a2ada4814ac7\") " Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186794 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.186920 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.187547 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.187568 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.187582 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.187594 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63478f6-c484-428b-bf9c-a2ada4814ac7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.187648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.187606 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.188936 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.188820 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:40.189191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.189108 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:40.189191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.189168 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:40.189623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.189599 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d" pod "e63478f6-c484-428b-bf9c-a2ada4814ac7" (UID: "e63478f6-c484-428b-bf9c-a2ada4814ac7"). InnerVolumeSpecName "kube-api-access-5w68d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:40.242121 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.242082 5991 generic.go:356] "Generic (PLEG): container finished" podID="e63478f6-c484-428b-bf9c-a2ada4814ac7" containerID="f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6" exitCode=0 Oct 30 19:54:40.242318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.242158 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" Oct 30 19:54:40.242318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.242178 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" event={"ID":"e63478f6-c484-428b-bf9c-a2ada4814ac7","Type":"ContainerDied","Data":"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6"} Oct 30 19:54:40.242318 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.242305 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf" event={"ID":"e63478f6-c484-428b-bf9c-a2ada4814ac7","Type":"ContainerDied","Data":"3c7c74e1bde6313b9f5acacc0a07c0f010fc3aa20e6819d27d3e29a6d13ed373"} Oct 30 19:54:40.242488 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.242333 5991 scope.go:122] "RemoveContainer" containerID="f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6" Oct 30 19:54:40.257026 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.256995 5991 scope.go:122] "RemoveContainer" containerID="f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6" Oct 30 19:54:40.257405 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:40.257334 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6\": container with ID starting with f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6 not found: ID does not exist" containerID="f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6" Oct 30 19:54:40.257505 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.257412 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6"} err="failed to get container status \"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6\": rpc error: code = NotFound desc = could not find container \"f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6\": container with ID starting with f2bb30fc702e04f6e3dbfcb98f39f758392905cd85c51499055b14e4bb1512d6 not found: ID does not exist" Oct 30 19:54:40.290194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.290160 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.290194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.290192 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.290461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.290205 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w68d\" (UniqueName: \"kubernetes.io/projected/e63478f6-c484-428b-bf9c-a2ada4814ac7-kube-api-access-5w68d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.290461 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.290224 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63478f6-c484-428b-bf9c-a2ada4814ac7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:54:40.294327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.294297 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:54:40.298526 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.298496 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c9c6d81d-openshift-default-66597449c7-mjcsf"] Oct 30 19:54:40.680178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:40.680145 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63478f6-c484-428b-bf9c-a2ada4814ac7" path="/var/lib/kubelet/pods/e63478f6-c484-428b-bf9c-a2ada4814ac7/volumes" Oct 30 19:54:49.634389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:49.622454 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lq4zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:54:49.634389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:49.622537 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.872290 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:54:50.872250 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:54:54.845354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:54.845301 5991 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podac4e089c-f524-46b3-a980-98946ec440d6"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podac4e089c-f524-46b3-a980-98946ec440d6] : Timed out while waiting for systemd to remove kubepods-burstable-podac4e089c_f524_46b3_a980_98946ec440d6.slice" Oct 30 19:54:54.845354 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:54.845348 5991 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="ac4e089c-f524-46b3-a980-98946ec440d6" err="failed to delete cgroup paths for [kubepods burstable podac4e089c-f524-46b3-a980-98946ec440d6] : unable to destroy cgroup paths for cgroup [kubepods burstable podac4e089c-f524-46b3-a980-98946ec440d6] : Timed out while waiting for systemd to remove kubepods-burstable-podac4e089c_f524_46b3_a980_98946ec440d6.slice" Oct 30 19:54:56.471402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:56.471348 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:54:56.510054 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:54:56.510020 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:55:01.472103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:01.472043 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" podUID="aa447404-f8f6-45d8-9821-6d855aba154d" containerName="payload-processing" containerID="cri-o://35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4" gracePeriod=25 Oct 30 19:55:01.511589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:01.511299 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" podUID="3d46d994-5f56-45ec-9395-77ab919b0f5c" containerName="payload-pre-processing" containerID="cri-o://46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f" gracePeriod=25 Oct 30 19:55:01.971886 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:01.971856 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:55:02.052042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.052004 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\" (UniqueName: \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\") pod \"aa447404-f8f6-45d8-9821-6d855aba154d\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " Oct 30 19:55:02.052233 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.052094 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\") pod \"aa447404-f8f6-45d8-9821-6d855aba154d\" (UID: \"aa447404-f8f6-45d8-9821-6d855aba154d\") " Oct 30 19:55:02.052601 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.052573 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume" pod "aa447404-f8f6-45d8-9821-6d855aba154d" (UID: "aa447404-f8f6-45d8-9821-6d855aba154d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:02.053286 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.053262 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa447404-f8f6-45d8-9821-6d855aba154d-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:55:02.054547 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.054487 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf" pod "aa447404-f8f6-45d8-9821-6d855aba154d" (UID: "aa447404-f8f6-45d8-9821-6d855aba154d"). InnerVolumeSpecName "kube-api-access-pr9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:02.063900 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.063872 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:55:02.154174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.154141 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\") pod \"3d46d994-5f56-45ec-9395-77ab919b0f5c\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " Oct 30 19:55:02.154174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.154187 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\" (UniqueName: \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\") pod \"3d46d994-5f56-45ec-9395-77ab919b0f5c\" (UID: \"3d46d994-5f56-45ec-9395-77ab919b0f5c\") " Oct 30 19:55:02.154647 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.154620 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume" pod "3d46d994-5f56-45ec-9395-77ab919b0f5c" (UID: "3d46d994-5f56-45ec-9395-77ab919b0f5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:02.155228 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.155203 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9nf\" (UniqueName: \"kubernetes.io/projected/aa447404-f8f6-45d8-9821-6d855aba154d-kube-api-access-pr9nf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:55:02.155330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.155233 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46d994-5f56-45ec-9395-77ab919b0f5c-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:55:02.156674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.156652 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp" pod "3d46d994-5f56-45ec-9395-77ab919b0f5c" (UID: "3d46d994-5f56-45ec-9395-77ab919b0f5c"). InnerVolumeSpecName "kube-api-access-kbqmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:02.257397 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.257341 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqmp\" (UniqueName: \"kubernetes.io/projected/3d46d994-5f56-45ec-9395-77ab919b0f5c-kube-api-access-kbqmp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:55:02.584512 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.584470 5991 generic.go:356] "Generic (PLEG): container finished" podID="aa447404-f8f6-45d8-9821-6d855aba154d" containerID="35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4" exitCode=0 Oct 30 19:55:02.584982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.584525 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" Oct 30 19:55:02.584982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.584536 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" event={"ID":"aa447404-f8f6-45d8-9821-6d855aba154d","Type":"ContainerDied","Data":"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4"} Oct 30 19:55:02.584982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.584593 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg" event={"ID":"aa447404-f8f6-45d8-9821-6d855aba154d","Type":"ContainerDied","Data":"68977c1df76d60b96aee8999addd90e0a47c98044c62777efbfdc545bb7c5c27"} Oct 30 19:55:02.584982 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.584621 5991 scope.go:122] "RemoveContainer" containerID="35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4" Oct 30 19:55:02.586099 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.586049 5991 generic.go:356] "Generic (PLEG): container finished" podID="3d46d994-5f56-45ec-9395-77ab919b0f5c" containerID="46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f" exitCode=0 Oct 30 19:55:02.586176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.586158 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" event={"ID":"3d46d994-5f56-45ec-9395-77ab919b0f5c","Type":"ContainerDied","Data":"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f"} Oct 30 19:55:02.586231 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.586199 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" event={"ID":"3d46d994-5f56-45ec-9395-77ab919b0f5c","Type":"ContainerDied","Data":"1ea6b5963996fc4534e2c8fe373be35c927e9386ccc50370a90e277f169d4da0"} Oct 30 19:55:02.586283 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.586262 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v" Oct 30 19:55:02.597583 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.597562 5991 scope.go:122] "RemoveContainer" containerID="35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4" Oct 30 19:55:02.597995 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:55:02.597972 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4\": container with ID starting with 35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4 not found: ID does not exist" containerID="35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4" Oct 30 19:55:02.598111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.598001 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4"} err="failed to get container status \"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4\": rpc error: code = NotFound desc = could not find container \"35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4\": container with ID starting with 35e3716c686772faf7a76c7bbb9ab2df4a2f667abebb22fe40d7cd53295a68a4 not found: ID does not exist" Oct 30 19:55:02.598111 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.598023 5991 scope.go:122] "RemoveContainer" containerID="46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f" Oct 30 19:55:02.608864 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.608845 5991 scope.go:122] "RemoveContainer" containerID="46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f" Oct 30 19:55:02.609172 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:55:02.609155 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f\": container with ID starting with 46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f not found: ID does not exist" containerID="46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f" Oct 30 19:55:02.609284 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.609185 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f"} err="failed to get container status \"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f\": rpc error: code = NotFound desc = could not find container \"46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f\": container with ID starting with 46d32c7cba569bbee7d7a5804cb4a726a864fdc052cbd47493b769f7322ec82f not found: ID does not exist" Oct 30 19:55:02.646731 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.646698 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:55:02.650419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.650391 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-75082b11-5757fbd9d4-7bwgg"] Oct 30 19:55:02.681457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.681422 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa447404-f8f6-45d8-9821-6d855aba154d" path="/var/lib/kubelet/pods/aa447404-f8f6-45d8-9821-6d855aba154d/volumes" Oct 30 19:55:02.682154 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.682132 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:55:02.691479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:02.691440 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-75082b11-57b5b6b99f-z4c8v"] Oct 30 19:55:04.682469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:04.682433 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d46d994-5f56-45ec-9395-77ab919b0f5c" path="/var/lib/kubelet/pods/3d46d994-5f56-45ec-9395-77ab919b0f5c/volumes" Oct 30 19:55:08.212306 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:08.210249 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:55:19.765084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.765046 5991 generic.go:356] "Generic (PLEG): container finished" podID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerID="bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6" exitCode=0 Oct 30 19:55:19.765660 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.765121 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerDied","Data":"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6"} Oct 30 19:55:19.765830 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.765813 5991 scope.go:122] "RemoveContainer" containerID="bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6" Oct 30 19:55:19.767183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.767158 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:19.767242 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.767228 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:19.767277 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:19.767261 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:20.796726 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:20.796682 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerStarted","Data":"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380"} Oct 30 19:55:20.895760 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:55:20.895726 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:55:21.125645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:21.125552 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:55:21.125937 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:21.125911 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:21.126050 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:21.125967 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:22.125924 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:22.125888 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:22.126331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:22.125953 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:23.125250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:23.125213 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:23.125444 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:23.125277 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:24.125238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:24.125206 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:24.125749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:24.125276 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:25.125759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:25.125722 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:25.126218 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:25.125784 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:26.125723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:26.125687 5991 patch_prober.go:29] interesting pod/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:55:26.125903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:26.125751 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 19:55:27.129961 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:27.129927 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:55:27.130343 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:27.130215 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:55:27.131139 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:27.131116 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:55:32.898788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:32.898754 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:55:32.906615 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:32.906593 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 19:55:50.869522 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:55:50.869480 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:55:53.987987 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.987948 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 19:55:53.988776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988758 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e63478f6-c484-428b-bf9c-a2ada4814ac7" containerName="istio-proxy" Oct 30 19:55:53.988842 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988777 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63478f6-c484-428b-bf9c-a2ada4814ac7" containerName="istio-proxy" Oct 30 19:55:53.988842 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988798 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3d46d994-5f56-45ec-9395-77ab919b0f5c" containerName="payload-pre-processing" Oct 30 19:55:53.988842 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988804 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d46d994-5f56-45ec-9395-77ab919b0f5c" containerName="payload-pre-processing" Oct 30 19:55:53.988943 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988850 5991 cpu_manager.go:414] "containerMap: removing container" podUID="aa447404-f8f6-45d8-9821-6d855aba154d" containerName="payload-processing" Oct 30 19:55:53.988943 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988855 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa447404-f8f6-45d8-9821-6d855aba154d" containerName="payload-processing" Oct 30 19:55:53.989004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988950 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa447404-f8f6-45d8-9821-6d855aba154d" containerName="payload-processing" Oct 30 19:55:53.989004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988961 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63478f6-c484-428b-bf9c-a2ada4814ac7" containerName="istio-proxy" Oct 30 19:55:53.989004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.988970 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d46d994-5f56-45ec-9395-77ab919b0f5c" containerName="payload-pre-processing" Oct 30 19:55:53.991586 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:53.991569 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.004440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.004402 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c8fa69-serving-cert\"" Oct 30 19:55:54.018650 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.018616 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 19:55:54.111964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.111921 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.112137 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.112061 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.112307 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.112291 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.112402 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.112319 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46flb\" (UniqueName: \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.213698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.213667 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.213888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.213734 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.213939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.213905 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.213939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.213928 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46flb\" (UniqueName: \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.214557 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.214529 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.216205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.216180 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.216306 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.216245 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.233096 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.233068 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46flb\" (UniqueName: \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\") pod \"maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.309919 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.309836 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:54.635667 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:54.635638 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 19:55:54.636126 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:55:54.636102 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e5be3f_9f14_4ddb_b422_545ad1a62814.slice/crio-b5b916a03e7380e8ea0e53599238201086fa050b9774552a005f973e260be10a WatchSource:0}: Error finding container b5b916a03e7380e8ea0e53599238201086fa050b9774552a005f973e260be10a: Status 404 returned error can't find the container with id b5b916a03e7380e8ea0e53599238201086fa050b9774552a005f973e260be10a Oct 30 19:55:55.157098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:55.156742 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" event={"ID":"a1e5be3f-9f14-4ddb-b422-545ad1a62814","Type":"ContainerStarted","Data":"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee"} Oct 30 19:55:55.157098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:55.156815 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" event={"ID":"a1e5be3f-9f14-4ddb-b422-545ad1a62814","Type":"ContainerStarted","Data":"b5b916a03e7380e8ea0e53599238201086fa050b9774552a005f973e260be10a"} Oct 30 19:55:55.157098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:55.156983 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:55:55.205042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:55.204952 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" podStartSLOduration=1.873444431 podStartE2EDuration="2.204932197s" totalImagesPullingTime="331.487766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:55.199070883 +0000 UTC m=+1222.648563609" watchObservedRunningTime="2026-10-30 19:55:55.204932197 +0000 UTC m=+1222.654424924" Oct 30 19:55:58.276674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:55:58.276605 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:56:01.171563 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:01.171528 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 19:56:19.356533 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.356487 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 19:56:19.362500 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.362468 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.391457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.391419 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 19:56:19.391648 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.391499 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-21c4ab-openshift-default-dockercfg-kqhk6\"" Oct 30 19:56:19.527673 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.517624 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.555786 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.555740 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.556112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.556086 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.556384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.556343 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.556565 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.556548 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.556754 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.556739 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.575397 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.557098 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb5m\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.575892 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.575858 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.576144 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.576127 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.684735 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685273 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685311 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685400 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685479 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685694 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb5m\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685781 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685838 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685898 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.685221 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.686688 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.686934 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.688207 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.690724 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.688687 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.696017 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.695985 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.712118 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.711716 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb5m\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.712650 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.712621 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.718076 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.718040 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\") pod \"e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:19.981754 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:19.981718 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:20.331439 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:56:20.330105 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8debdeb_b068_458e_866b_4246249c3b52.slice/crio-2a966f8ecd2e4bc2bd956d5a3ab75cbda62f7f1eee4b11eb12dd279e78a4ad4c WatchSource:0}: Error finding container 2a966f8ecd2e4bc2bd956d5a3ab75cbda62f7f1eee4b11eb12dd279e78a4ad4c: Status 404 returned error can't find the container with id 2a966f8ecd2e4bc2bd956d5a3ab75cbda62f7f1eee4b11eb12dd279e78a4ad4c Oct 30 19:56:20.331439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:20.330143 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 19:56:20.334028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:20.333991 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:20.334164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:20.334066 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:20.334164 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:20.334102 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:20.374936 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:20.374896 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerStarted","Data":"2a966f8ecd2e4bc2bd956d5a3ab75cbda62f7f1eee4b11eb12dd279e78a4ad4c"} Oct 30 19:56:20.871897 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:56:20.871863 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:56:21.384219 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:21.384176 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerStarted","Data":"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087"} Oct 30 19:56:21.444556 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:21.444493 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podStartSLOduration=2.444478959 podStartE2EDuration="2.444478959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:21.442588593 +0000 UTC m=+1248.892081320" watchObservedRunningTime="2026-10-30 19:56:21.444478959 +0000 UTC m=+1248.893971684" Oct 30 19:56:21.982424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:21.982388 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:21.989151 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:21.989118 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:22.391527 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:22.391440 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:22.392494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:22.392475 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:56:24.655477 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.654647 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 19:56:24.668042 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.668006 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.732573 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.732439 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5c2e32\"" Oct 30 19:56:24.755659 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.755624 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5c2e32-dockercfg-j72wc\"" Oct 30 19:56:24.821482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.820973 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5lw\" (UniqueName: \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.823414 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.823028 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.860929 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.860876 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 19:56:24.927510 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.927353 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.927934 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.927902 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5lw\" (UniqueName: \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.928330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.928311 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:24.948159 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:24.948123 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5lw\" (UniqueName: \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\") pod \"payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:25.092845 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:25.092806 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:25.413390 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:56:25.413345 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c6216b_e0b5_4858_8bc7_72793593fa13.slice/crio-bad885ee752aa646c696339e147c3a6a0f29ac937db2d0e6b67a18aeee1160b8 WatchSource:0}: Error finding container bad885ee752aa646c696339e147c3a6a0f29ac937db2d0e6b67a18aeee1160b8: Status 404 returned error can't find the container with id bad885ee752aa646c696339e147c3a6a0f29ac937db2d0e6b67a18aeee1160b8 Oct 30 19:56:25.414423 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:25.414398 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 19:56:26.250794 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.250754 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 19:56:26.255800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.255767 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.273995 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.273721 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-21c4ab\"" Oct 30 19:56:26.287480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.287447 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 19:56:26.324484 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.324446 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 19:56:26.330671 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.330602 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.355459 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.346784 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-21c4ab-dockercfg-6qsgn\"" Oct 30 19:56:26.359939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.359495 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.360090 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.359993 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbt7w\" (UniqueName: \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.363569 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.361403 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 19:56:26.477390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.477129 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.477390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.477375 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbt7w\" (UniqueName: \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.477657 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.477466 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.477657 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.477495 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb82c\" (UniqueName: \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.488463 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.479224 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.519449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.499046 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" event={"ID":"d3c6216b-e0b5-4858-8bc7-72793593fa13","Type":"ContainerStarted","Data":"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a"} Oct 30 19:56:26.519449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.499115 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" event={"ID":"d3c6216b-e0b5-4858-8bc7-72793593fa13","Type":"ContainerStarted","Data":"bad885ee752aa646c696339e147c3a6a0f29ac937db2d0e6b67a18aeee1160b8"} Oct 30 19:56:26.519449 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.500355 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:26.530744 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.530708 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbt7w\" (UniqueName: \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\") pod \"payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.555394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.551883 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" podStartSLOduration=2.551862569 podStartE2EDuration="2.551862569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:26.550538321 +0000 UTC m=+1254.000031098" watchObservedRunningTime="2026-10-30 19:56:26.551862569 +0000 UTC m=+1254.001355297" Oct 30 19:56:26.573194 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.573153 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:26.579909 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.579867 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb82c\" (UniqueName: \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.580171 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.580134 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.580893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.580870 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.602887 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.602852 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb82c\" (UniqueName: \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\") pod \"payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:26.689188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:26.689147 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:27.232385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.231567 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 19:56:27.253782 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.251900 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 19:56:27.511068 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.511015 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" event={"ID":"d7754cee-5ad8-4cc3-8fbd-751ecb155b51","Type":"ContainerStarted","Data":"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca"} Oct 30 19:56:27.511325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.511128 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" event={"ID":"d7754cee-5ad8-4cc3-8fbd-751ecb155b51","Type":"ContainerStarted","Data":"5ebaf62b24283bca50304839671211b575edf1213e1f2dfc0cc80fcddeff0c97"} Oct 30 19:56:27.511325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.511201 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:27.514245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.514205 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" event={"ID":"137b2e28-6d44-4676-81fd-5918945a4f1e","Type":"ContainerStarted","Data":"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac"} Oct 30 19:56:27.514421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.514271 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" event={"ID":"137b2e28-6d44-4676-81fd-5918945a4f1e","Type":"ContainerStarted","Data":"9de15b3eacc4e6b60350d0dc3aa8501231dc24c63b56b0095fa9c034d9475756"} Oct 30 19:56:27.514906 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.514881 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:27.567177 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.567111 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" podStartSLOduration=1.567088848 podStartE2EDuration="1.567088848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:27.55270654 +0000 UTC m=+1255.002199265" watchObservedRunningTime="2026-10-30 19:56:27.567088848 +0000 UTC m=+1255.016581574" Oct 30 19:56:27.618204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:27.618140 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" podStartSLOduration=1.6181219439999999 podStartE2EDuration="1.618121944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:27.603686789 +0000 UTC m=+1255.053179522" watchObservedRunningTime="2026-10-30 19:56:27.618121944 +0000 UTC m=+1255.067614669" Oct 30 19:56:38.526069 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:38.526038 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 19:56:38.526491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:38.526116 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 19:56:38.526491 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:38.526156 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 19:56:45.489598 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.489516 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 19:56:45.496535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.496504 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.512352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.512319 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-5c2e32-serving-cert\"" Oct 30 19:56:45.533668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.533631 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.533849 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.533691 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.534046 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.534020 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rsr6\" (UniqueName: \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.534131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.534092 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.534420 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.534399 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 19:56:45.636414 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.636324 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rsr6\" (UniqueName: \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.636636 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.636430 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.636636 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.636576 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.636636 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:56:45.636581 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-serving-cert: secret "maas-api-e2e-shared-a-w6-5c2e32-serving-cert" not found Oct 30 19:56:45.636636 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.636612 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.636865 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:56:45.636670 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls podName:3b25294d-fe92-4a3e-becc-db8ef0331beb nodeName:}" failed. No retries permitted until 2026-10-30 19:56:46.136646589 +0000 UTC m=+1273.586139304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls") pod "maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" (UID: "3b25294d-fe92-4a3e-becc-db8ef0331beb") : secret "maas-api-e2e-shared-a-w6-5c2e32-serving-cert" not found Oct 30 19:56:45.637557 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.637528 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.645418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.640040 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:45.664471 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:45.664436 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rsr6\" (UniqueName: \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:46.166394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:46.165695 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:46.219394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:46.202966 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:46.418930 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:46.418826 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:46.751992 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:56:46.751951 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b25294d_fe92_4a3e_becc_db8ef0331beb.slice/crio-c85a159d4bdf3458cc1e9d8f874e098b059393bd105fc921278c1589476f2a0c WatchSource:0}: Error finding container c85a159d4bdf3458cc1e9d8f874e098b059393bd105fc921278c1589476f2a0c: Status 404 returned error can't find the container with id c85a159d4bdf3458cc1e9d8f874e098b059393bd105fc921278c1589476f2a0c Oct 30 19:56:46.753020 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:46.752995 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 19:56:47.709046 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:47.709010 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" event={"ID":"3b25294d-fe92-4a3e-becc-db8ef0331beb","Type":"ContainerStarted","Data":"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859"} Oct 30 19:56:47.709238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:47.709072 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:47.709238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:47.709094 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" event={"ID":"3b25294d-fe92-4a3e-becc-db8ef0331beb","Type":"ContainerStarted","Data":"c85a159d4bdf3458cc1e9d8f874e098b059393bd105fc921278c1589476f2a0c"} Oct 30 19:56:47.767804 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:47.767745 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" podStartSLOduration=2.394267956 podStartE2EDuration="2.76772875s" totalImagesPullingTime="373.460794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:47.751011273 +0000 UTC m=+1275.200503998" watchObservedRunningTime="2026-10-30 19:56:47.76772875 +0000 UTC m=+1275.217221491" Oct 30 19:56:47.885220 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:47.885139 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:56:50.050707 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.050667 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 19:56:50.057442 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.057418 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.072224 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.072192 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-21c4ab-serving-cert\"" Oct 30 19:56:50.089739 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.089707 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 19:56:50.134940 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.134906 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.135132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.135111 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.135335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.135314 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcctj\" (UniqueName: \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.135470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.135455 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.237869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.237831 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.238095 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.238074 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcctj\" (UniqueName: \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.238237 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.238221 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.238284 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.238269 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.239064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.239033 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.240696 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.240672 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.240807 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.240755 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.259178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.259143 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcctj\" (UniqueName: \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\") pod \"maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.374827 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.374739 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:50.892190 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:50.890912 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 19:56:50.937314 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:56:50.937262 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:56:51.900635 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:51.900593 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" event={"ID":"b98ff840-e3a2-44e3-9e14-3881d883dbcb","Type":"ContainerStarted","Data":"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d"} Oct 30 19:56:51.901201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:51.900659 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" event={"ID":"b98ff840-e3a2-44e3-9e14-3881d883dbcb","Type":"ContainerStarted","Data":"5f3b724f7834643711ee8b6c2d9bfe844a9dd2aac2a7048b1fafd22cd5187861"} Oct 30 19:56:51.901201 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:51.900993 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:56:51.960331 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:51.960269 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" podStartSLOduration=1.587823576 podStartE2EDuration="1.96024986s" totalImagesPullingTime="372.426284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:51.943963214 +0000 UTC m=+1279.393455939" watchObservedRunningTime="2026-10-30 19:56:51.96024986 +0000 UTC m=+1279.409742585" Oct 30 19:56:53.725255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.725227 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 19:56:53.875404 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.875318 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:56:53.881100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.881070 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:53.897126 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.896722 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8d97a5\"" Oct 30 19:56:53.899762 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.899722 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:56:53.906165 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.906136 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8d97a5-dockercfg-tmv6m\"" Oct 30 19:56:53.994858 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.994753 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:53.996346 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:53.994881 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkhn\" (UniqueName: \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.098176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.097556 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.098176 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.097715 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkhn\" (UniqueName: \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.099217 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.099181 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.138268 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.137115 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkhn\" (UniqueName: \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\") pod \"payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.200282 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.199401 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.532579 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:56:54.532542 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021986b0_ee75_44ea_90b7_96752a2133de.slice/crio-d271e3502ac18bcf806e26e1d03273c264c710c818b149166c68a24c75a9cdc1 WatchSource:0}: Error finding container d271e3502ac18bcf806e26e1d03273c264c710c818b149166c68a24c75a9cdc1: Status 404 returned error can't find the container with id d271e3502ac18bcf806e26e1d03273c264c710c818b149166c68a24c75a9cdc1 Oct 30 19:56:54.535135 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.535097 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:56:54.933171 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.933064 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" event={"ID":"021986b0-ee75-44ea-90b7-96752a2133de","Type":"ContainerStarted","Data":"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254"} Oct 30 19:56:54.933171 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.933139 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" event={"ID":"021986b0-ee75-44ea-90b7-96752a2133de","Type":"ContainerStarted","Data":"d271e3502ac18bcf806e26e1d03273c264c710c818b149166c68a24c75a9cdc1"} Oct 30 19:56:54.933717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.933496 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:56:54.973576 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:54.973515 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" podStartSLOduration=1.973500514 podStartE2EDuration="1.973500514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:54.971847597 +0000 UTC m=+1282.421340322" watchObservedRunningTime="2026-10-30 19:56:54.973500514 +0000 UTC m=+1282.422993242" Oct 30 19:56:57.915468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:56:57.915432 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 19:57:05.945303 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:05.945271 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:57:17.485515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:17.485451 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:57:17.491769 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:17.489332 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" containerID="cri-o://bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380" gracePeriod=30 Oct 30 19:57:20.882594 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:20.882506 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:57:21.352924 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.352884 5991 generic.go:356] "Generic (PLEG): container finished" podID="a8debdeb-b068-458e-866b-4246249c3b52" containerID="f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087" exitCode=0 Oct 30 19:57:21.353144 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.352954 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerDied","Data":"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087"} Oct 30 19:57:21.353521 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.353494 5991 scope.go:122] "RemoveContainer" containerID="f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087" Oct 30 19:57:21.355714 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.355681 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:21.355793 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.355779 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:21.355847 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:21.355831 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:22.370428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.367734 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerStarted","Data":"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064"} Oct 30 19:57:22.392166 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.392132 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:57:22.786117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.786080 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:57:22.896749 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.896699 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.896957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.896809 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.896957 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.896931 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897082 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.896994 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:22.897082 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897006 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897199 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897127 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897291 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897268 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897288 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:22.897384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897328 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897422 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897471 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\") pod \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\" (UID: \"ba8e8ce5-3da1-45ba-9623-72abd65ed82e\") " Oct 30 19:57:22.897607 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897536 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:22.897857 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897836 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:22.898038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.897862 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:22.898502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.898482 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:22.898623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.898612 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:22.898711 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.898701 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:22.898785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.898775 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:22.898862 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.898853 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:22.900718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.900254 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "kube-api-access-995t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:22.900718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.900450 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:22.900718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.900643 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:22.902343 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.902312 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy" pod "ba8e8ce5-3da1-45ba-9623-72abd65ed82e" (UID: "ba8e8ce5-3da1-45ba-9623-72abd65ed82e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:22.994473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.983395 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:57:22.994473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.984450 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:22.994473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:22.984509 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:23.001783 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.001745 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:23.001783 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.001783 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:23.002005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.001798 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995t7\" (UniqueName: \"kubernetes.io/projected/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-kube-api-access-995t7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:23.002005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.001815 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e8ce5-3da1-45ba-9623-72abd65ed82e-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:57:23.399186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.399088 5991 generic.go:356] "Generic (PLEG): container finished" podID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerID="bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380" exitCode=0 Oct 30 19:57:23.399186 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.399165 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" Oct 30 19:57:23.399723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.399164 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerDied","Data":"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380"} Oct 30 19:57:23.399723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.399284 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b" event={"ID":"ba8e8ce5-3da1-45ba-9623-72abd65ed82e","Type":"ContainerDied","Data":"18f1768a2c846a127d60b29aaa67f350bb6098103424a669f463d949d9571ecf"} Oct 30 19:57:23.399723 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.399307 5991 scope.go:122] "RemoveContainer" containerID="bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380" Oct 30 19:57:23.411700 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.411680 5991 scope.go:122] "RemoveContainer" containerID="bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6" Oct 30 19:57:23.422428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.422405 5991 scope.go:122] "RemoveContainer" containerID="bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380" Oct 30 19:57:23.422775 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:23.422753 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380\": container with ID starting with bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380 not found: ID does not exist" containerID="bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380" Oct 30 19:57:23.422831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.422787 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380"} err="failed to get container status \"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380\": rpc error: code = NotFound desc = could not find container \"bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380\": container with ID starting with bd4820c1043853c9a88e7d462d88da4638a506b0114e4e5be7f7b090b5218380 not found: ID does not exist" Oct 30 19:57:23.422831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.422807 5991 scope.go:122] "RemoveContainer" containerID="bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6" Oct 30 19:57:23.423070 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:23.423052 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6\": container with ID starting with bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6 not found: ID does not exist" containerID="bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6" Oct 30 19:57:23.423126 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.423080 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6"} err="failed to get container status \"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6\": rpc error: code = NotFound desc = could not find container \"bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6\": container with ID starting with bf1415a458d3c5597afb6faebe0f07c5c248bc99436c0eb589b2a2af4d6d0eb6 not found: ID does not exist" Oct 30 19:57:23.496588 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.496546 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:57:23.503865 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.503825 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c53b7887-openshift-default-74d99bd4d9-7g67b"] Oct 30 19:57:23.988531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.986811 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:23.988531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:23.986873 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:24.143568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.143525 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 19:57:24.145105 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.145075 5991 cpu_manager.go:414] "containerMap: removing container" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:24.145105 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.145101 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:24.145393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.145298 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:24.145393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.145311 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:24.150977 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.150952 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.202105 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.202076 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-93263c-serving-cert\"" Oct 30 19:57:24.211827 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.211783 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 19:57:24.324539 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.324447 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.324696 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.324611 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w66p\" (UniqueName: \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.324750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.324735 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.324871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.324856 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.426243 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.426200 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w66p\" (UniqueName: \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.426822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.426341 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.426822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.426506 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.426822 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.426681 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.427556 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.427531 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.429087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.429061 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.429209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.429061 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.447948 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.447911 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w66p\" (UniqueName: \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\") pod \"maas-api-e2e-api-a-93263c-6b9696fff5-r249g\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.502669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.502628 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:24.756394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.753708 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" path="/var/lib/kubelet/pods/ba8e8ce5-3da1-45ba-9623-72abd65ed82e/volumes" Oct 30 19:57:24.987424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.985813 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:24.987424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:24.985883 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:25.124038 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:25.124013 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 19:57:25.418602 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:25.418497 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" event={"ID":"3c3280a4-8496-4a2e-bc07-8d11b0de51fe","Type":"ContainerStarted","Data":"dd10eabffb9a2c560f7d1279be50c28f00a089baac533884e5ec72f36540e92b"} Oct 30 19:57:25.984444 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:25.982813 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:25.984444 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:25.982877 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:26.433460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:26.431198 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" event={"ID":"3c3280a4-8496-4a2e-bc07-8d11b0de51fe","Type":"ContainerStarted","Data":"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17"} Oct 30 19:57:26.433460 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:26.431328 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:26.546396 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:26.540060 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" podStartSLOduration=2.14382814 podStartE2EDuration="2.54004041s" totalImagesPullingTime="396.21227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:26.517682788 +0000 UTC m=+1313.967175513" watchObservedRunningTime="2026-10-30 19:57:26.54004041 +0000 UTC m=+1313.989533135" Oct 30 19:57:26.986383 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:26.986323 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:26.986985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:26.986948 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:27.990384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:27.989786 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:57:27.990384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:27.989855 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:57:28.987132 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:28.987098 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:57:28.988315 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:28.988288 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 19:57:32.524656 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:32.524628 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 19:57:50.870572 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:50.870530 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:57:53.179384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.179317 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 19:57:53.180520 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.180496 5991 cpu_manager.go:414] "containerMap: removing container" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:53.180520 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.180518 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8e8ce5-3da1-45ba-9623-72abd65ed82e" containerName="istio-proxy" Oct 30 19:57:53.184710 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.184684 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.201489 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.201458 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d96913-openshift-default-dockercfg-hnpk2\"" Oct 30 19:57:53.220536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.220499 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 19:57:53.221181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221146 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221311 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221253 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221407 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221432 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221475 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221592 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221577 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8x4\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221633 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221625 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221641 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.221710 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.221673 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.345673 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.345624 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354482 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354571 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354691 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8x4\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354767 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354792 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.354847 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.355077 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.355131 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.355772 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.357394 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.346300 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.369400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.359636 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.369400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.360279 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.369400 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.360864 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.383457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.378967 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.434437 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.418001 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.434671 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.426783 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8x4\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.434780 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.432956 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\") pod \"e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.506178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.506139 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:53.855468 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:57:53.855436 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a764cab_894a_4a5d_9aa4_cad98acd9a69.slice/crio-d1455149a73040bd666046b550fa12552233394da583e840e69cc3eff9a464f8 WatchSource:0}: Error finding container d1455149a73040bd666046b550fa12552233394da583e840e69cc3eff9a464f8: Status 404 returned error can't find the container with id d1455149a73040bd666046b550fa12552233394da583e840e69cc3eff9a464f8 Oct 30 19:57:53.856684 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.856657 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 19:57:53.858502 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.858470 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:53.858591 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.858549 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:53.858591 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:53.858580 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:54.552935 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.552904 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 19:57:54.557877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.557858 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.575551 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.575456 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a165e5-serving-cert\"" Oct 30 19:57:54.580869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.580837 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 19:57:54.681776 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.681747 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.681952 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.681930 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.681999 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.681988 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.682078 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.682063 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.783775 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.783736 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.784288 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.783983 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.784288 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.784251 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.784543 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.784335 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.784543 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:54.784492 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-serving-cert: secret "maas-api-e2e-shared-a-w2-a165e5-serving-cert" not found Oct 30 19:57:54.784658 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:57:54.784576 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls podName:c3bc0c24-5f35-4f75-b22d-728604f7e6bb nodeName:}" failed. No retries permitted until 2026-10-30 19:57:55.284552549 +0000 UTC m=+1342.734045275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" (UID: "c3bc0c24-5f35-4f75-b22d-728604f7e6bb") : secret "maas-api-e2e-shared-a-w2-a165e5-serving-cert" not found Oct 30 19:57:54.785183 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.785135 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.792575 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.792540 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.807352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.807275 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:54.809702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.809588 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerStarted","Data":"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a"} Oct 30 19:57:54.809702 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.809655 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerStarted","Data":"d1455149a73040bd666046b550fa12552233394da583e840e69cc3eff9a464f8"} Oct 30 19:57:54.862063 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:54.861999 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podStartSLOduration=1.861978057 podStartE2EDuration="1.861978057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:54.859859852 +0000 UTC m=+1342.309352581" watchObservedRunningTime="2026-10-30 19:57:54.861978057 +0000 UTC m=+1342.311470784" Oct 30 19:57:55.293763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.293723 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:55.302991 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.302956 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:55.473739 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.473702 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:55.506926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.506831 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:55.518349 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.518319 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:55.789578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.789546 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 19:57:55.821869 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.821757 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" event={"ID":"c3bc0c24-5f35-4f75-b22d-728604f7e6bb","Type":"ContainerStarted","Data":"d4eb48e9cb8da39e66bba84a949ddef19808a99d00038ed90ee647084b8fb8fc"} Oct 30 19:57:55.822204 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.822180 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:55.823554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:55.823518 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 19:57:56.409536 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:56.406714 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:57:56.832720 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:56.832675 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" event={"ID":"c3bc0c24-5f35-4f75-b22d-728604f7e6bb","Type":"ContainerStarted","Data":"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e"} Oct 30 19:57:56.833191 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:56.832746 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:57:56.890917 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:57:56.890638 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" podStartSLOduration=2.562557994 podStartE2EDuration="2.89060958s" totalImagesPullingTime="328.051586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:56.881454143 +0000 UTC m=+1344.330946868" watchObservedRunningTime="2026-10-30 19:57:56.89060958 +0000 UTC m=+1344.340102306" Oct 30 19:58:00.662327 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.662287 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 19:58:00.668577 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.668550 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.706953 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.706914 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 19:58:00.735517 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.735437 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.737950 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.735489 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5br\" (UniqueName: \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.808158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.794968 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d96913-dockercfg-27jcm\"" Oct 30 19:58:00.808158 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.795082 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d96913\"" Oct 30 19:58:00.864385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.862842 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.864385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.862894 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5br\" (UniqueName: \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.882058 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.864763 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.979028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.978775 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5br\" (UniqueName: \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\") pod \"payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:00.988959 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:00.988839 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:01.362781 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:01.362748 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03908caf_8fc5_4f4d_9996_e9ae0868b912.slice/crio-68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4 WatchSource:0}: Error finding container 68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4: Status 404 returned error can't find the container with id 68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4 Oct 30 19:58:01.363997 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:01.363973 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 19:58:01.998467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:01.998425 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" event={"ID":"03908caf-8fc5-4f4d-9996-e9ae0868b912","Type":"ContainerStarted","Data":"46d948fac3df18530905239ec04732ed08781671515b0c2e76ccc7452e5c7b16"} Oct 30 19:58:01.998873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:01.998486 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" event={"ID":"03908caf-8fc5-4f4d-9996-e9ae0868b912","Type":"ContainerStarted","Data":"68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4"} Oct 30 19:58:01.998873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:01.998534 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:02.044571 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:02.044498 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" podStartSLOduration=2.044483661 podStartE2EDuration="2.044483661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.042357427 +0000 UTC m=+1349.491850163" watchObservedRunningTime="2026-10-30 19:58:02.044483661 +0000 UTC m=+1349.493976386" Oct 30 19:58:02.911083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:02.911052 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 19:58:07.666174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.666133 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 19:58:07.672921 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.672886 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.698871 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.698828 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 19:58:07.704481 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.704447 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-540a43-openshift-default-dockercfg-nwmtb\"" Oct 30 19:58:07.772625 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772577 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.772825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772676 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.772825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772698 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.772825 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772731 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k658\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.772989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772829 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.772989 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.772932 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.773084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.773005 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.773273 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.773252 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.773314 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.773298 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882415 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882545 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882596 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882622 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882659 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k658\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882697 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882796 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.882883 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.883145 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.883974 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.886396 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.889428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.886654 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.921467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.910799 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.921467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.913253 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.943401 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.938066 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k658\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.951415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.944946 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.951415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.946228 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.960970 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.959216 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\") pod \"e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:07.998387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:07.995925 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:08.447453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:08.446430 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 19:58:08.449494 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:08.449298 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b593a0d_cebc_4ce9_af1b_ccd5c70e8ea2.slice/crio-ba16499c1e553f6adf381f8d39a1552bba47f0d7c3d53b581561f48bfbaad7c6 WatchSource:0}: Error finding container ba16499c1e553f6adf381f8d39a1552bba47f0d7c3d53b581561f48bfbaad7c6: Status 404 returned error can't find the container with id ba16499c1e553f6adf381f8d39a1552bba47f0d7c3d53b581561f48bfbaad7c6 Oct 30 19:58:08.456048 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:08.455994 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:08.456239 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:08.456131 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:08.456239 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:08.456190 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:09.108045 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:09.108003 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerStarted","Data":"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1"} Oct 30 19:58:09.108469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:09.108068 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerStarted","Data":"ba16499c1e553f6adf381f8d39a1552bba47f0d7c3d53b581561f48bfbaad7c6"} Oct 30 19:58:09.183729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:09.183671 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podStartSLOduration=2.183655154 podStartE2EDuration="2.183655154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:09.169684254 +0000 UTC m=+1356.619176979" watchObservedRunningTime="2026-10-30 19:58:09.183655154 +0000 UTC m=+1356.633147878" Oct 30 19:58:09.996771 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:09.996731 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:10.002949 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:10.002918 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:10.115300 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:10.115267 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:10.116507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:10.116478 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:58:13.011769 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.011737 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 19:58:13.453245 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.453155 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 19:58:13.460994 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.459621 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.476092 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.476061 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-540a43-dockercfg-wh79v\"" Oct 30 19:58:13.476922 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.476468 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-540a43\"" Oct 30 19:58:13.484469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.484435 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 19:58:13.526386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.522294 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfwh\" (UniqueName: \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.526386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.522493 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.629395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.625250 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfwh\" (UniqueName: \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.629395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.625471 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.629395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.626747 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.688517 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.688426 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 19:58:13.731056 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.731010 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.773248 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.773164 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfwh\" (UniqueName: \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\") pod \"payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.786031 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.785996 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:13.832258 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.830710 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrhq\" (UniqueName: \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.832258 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.830838 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.865393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.864451 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 19:58:13.933117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.933071 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrhq\" (UniqueName: \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.933326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.933168 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.934246 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.934213 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:13.980806 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:13.980571 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrhq\" (UniqueName: \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\") pod \"payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:14.075796 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:14.075708 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:14.183759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:14.181136 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 19:58:14.397745 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:14.397708 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8bad11_f317_4812_bb49_26a34220c5b4.slice/crio-6a20f19857af55a70342d64cf578c7fc3504f070c054f5e250438d454140f375 WatchSource:0}: Error finding container 6a20f19857af55a70342d64cf578c7fc3504f070c054f5e250438d454140f375: Status 404 returned error can't find the container with id 6a20f19857af55a70342d64cf578c7fc3504f070c054f5e250438d454140f375 Oct 30 19:58:14.398554 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:14.398528 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 19:58:15.176531 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.176488 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" event={"ID":"fe8bad11-f317-4812-bb49-26a34220c5b4","Type":"ContainerStarted","Data":"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829"} Oct 30 19:58:15.176958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.176565 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:15.176958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.176590 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" event={"ID":"fe8bad11-f317-4812-bb49-26a34220c5b4","Type":"ContainerStarted","Data":"6a20f19857af55a70342d64cf578c7fc3504f070c054f5e250438d454140f375"} Oct 30 19:58:15.178879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.178843 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" event={"ID":"232954cc-e118-44ec-8e48-5f01f89837ee","Type":"ContainerStarted","Data":"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9"} Oct 30 19:58:15.179053 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.178901 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" event={"ID":"232954cc-e118-44ec-8e48-5f01f89837ee","Type":"ContainerStarted","Data":"6cb471a9cf05b964a8ae53e794771580a14719a1c15d992ee452339b7ca1d925"} Oct 30 19:58:15.179053 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.178929 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:15.219241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.219144 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" podStartSLOduration=2.219126015 podStartE2EDuration="2.219126015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:15.218098835 +0000 UTC m=+1362.667591564" watchObservedRunningTime="2026-10-30 19:58:15.219126015 +0000 UTC m=+1362.668618741" Oct 30 19:58:15.262903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:15.262850 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" podStartSLOduration=2.262834666 podStartE2EDuration="2.262834666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:15.262642463 +0000 UTC m=+1362.712135189" watchObservedRunningTime="2026-10-30 19:58:15.262834666 +0000 UTC m=+1362.712327392" Oct 30 19:58:20.870535 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:20.870496 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:58:26.191127 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:26.191095 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 19:58:26.191755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:26.191186 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 19:58:28.971174 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.971130 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:58:28.971718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.971458 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" containerID="cri-o://c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0" gracePeriod=30 Oct 30 19:58:28.986393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.986017 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:58:28.986393 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.986271 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" containerID="cri-o://a353b2118a782d482401de021b16555b1f4f9a6720e71a6e77bb12ee373ec0f6" gracePeriod=30 Oct 30 19:58:28.991915 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.991877 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:58:28.992137 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:28.992113 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" containerID="cri-o://129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53" gracePeriod=30 Oct 30 19:58:29.487698 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.487654 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:58:29.488015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.487932 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" containerID="cri-o://0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9" gracePeriod=30 Oct 30 19:58:29.493087 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.493051 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:58:29.493414 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.493307 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" containerID="cri-o://909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f" gracePeriod=30 Oct 30 19:58:29.500907 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.500870 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:58:29.501136 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:29.501106 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" containerID="cri-o://f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218" gracePeriod=30 Oct 30 19:58:34.315306 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.312113 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:58:34.332480 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.332452 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:58:34.402940 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.402908 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403124 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403003 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.403124 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403036 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403124 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403082 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.403418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403131 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403227 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.403418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403235 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403378 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403418 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403412 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403521 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.403716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403531 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.403716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403591 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403716 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403648 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.403908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403739 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.403908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403814 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.403908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403847 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.404061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.403927 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.404061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404014 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.404061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404051 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.404209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404061 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.404209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404090 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\") pod \"7239b6c0-a0b9-4040-b25d-ada66fd09937\" (UID: \"7239b6c0-a0b9-4040-b25d-ada66fd09937\") " Oct 30 19:58:34.404916 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404459 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.404916 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404861 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\") pod \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\" (UID: \"59bb1deb-41a4-450c-a1c8-83ebd000e1ed\") " Oct 30 19:58:34.405172 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.404950 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.405247 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.405170 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.405542 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.405495 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.405772 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.405746 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.406516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406490 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406523 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406539 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406554 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406567 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406609 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406581 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406611 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7239b6c0-a0b9-4040-b25d-ada66fd09937-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406627 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406641 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.406945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406489 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "kube-api-access-vjk2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.406945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.406715 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.407752 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.407725 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.409979 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.409945 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:34.412629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.412333 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.412629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.412458 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:34.412629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.412315 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.412629 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.412586 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt" pod "7239b6c0-a0b9-4040-b25d-ada66fd09937" (UID: "7239b6c0-a0b9-4040-b25d-ada66fd09937"). InnerVolumeSpecName "kube-api-access-mx6jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.424589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.424558 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token" pod "59bb1deb-41a4-450c-a1c8-83ebd000e1ed" (UID: "59bb1deb-41a4-450c-a1c8-83ebd000e1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.425747 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.425721 5991 generic.go:356] "Generic (PLEG): container finished" podID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerID="a353b2118a782d482401de021b16555b1f4f9a6720e71a6e77bb12ee373ec0f6" exitCode=0 Oct 30 19:58:34.425874 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.425797 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" event={"ID":"854bf2a9-4868-4b62-a01e-71c85911e4ff","Type":"ContainerDied","Data":"a353b2118a782d482401de021b16555b1f4f9a6720e71a6e77bb12ee373ec0f6"} Oct 30 19:58:34.427617 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.427588 5991 generic.go:356] "Generic (PLEG): container finished" podID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerID="129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53" exitCode=0 Oct 30 19:58:34.427750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.427697 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" event={"ID":"7239b6c0-a0b9-4040-b25d-ada66fd09937","Type":"ContainerDied","Data":"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53"} Oct 30 19:58:34.427750 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.427739 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" event={"ID":"7239b6c0-a0b9-4040-b25d-ada66fd09937","Type":"ContainerDied","Data":"ea602f5fef96560259803e6810cfd8281fefde11a632f0581836142bf448a466"} Oct 30 19:58:34.427872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.427768 5991 scope.go:122] "RemoveContainer" containerID="129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53" Oct 30 19:58:34.427872 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.427829 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx" Oct 30 19:58:34.430275 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.430247 5991 generic.go:356] "Generic (PLEG): container finished" podID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerID="c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0" exitCode=0 Oct 30 19:58:34.430428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.430289 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" event={"ID":"59bb1deb-41a4-450c-a1c8-83ebd000e1ed","Type":"ContainerDied","Data":"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0"} Oct 30 19:58:34.430428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.430322 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" Oct 30 19:58:34.430428 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.430329 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5" event={"ID":"59bb1deb-41a4-450c-a1c8-83ebd000e1ed","Type":"ContainerDied","Data":"31173ee40161421923efc5b9e52922f841c38d32cc619ce20ca342aa7b876b2b"} Oct 30 19:58:34.431974 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.431909 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:58:34.461493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.461468 5991 scope.go:122] "RemoveContainer" containerID="129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53" Oct 30 19:58:34.461860 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:34.461837 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53\": container with ID starting with 129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53 not found: ID does not exist" containerID="129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53" Oct 30 19:58:34.461964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.461869 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53"} err="failed to get container status \"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53\": rpc error: code = NotFound desc = could not find container \"129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53\": container with ID starting with 129f726a4270d9ec09273743f2d12642770e83bc825d7e878f9e663521f9de53 not found: ID does not exist" Oct 30 19:58:34.461964 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.461887 5991 scope.go:122] "RemoveContainer" containerID="c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0" Oct 30 19:58:34.472413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.472359 5991 scope.go:122] "RemoveContainer" containerID="c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0" Oct 30 19:58:34.472702 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:34.472684 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0\": container with ID starting with c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0 not found: ID does not exist" containerID="c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0" Oct 30 19:58:34.472792 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.472710 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0"} err="failed to get container status \"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0\": rpc error: code = NotFound desc = could not find container \"c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0\": container with ID starting with c36b62432c6ca43a433fd464b723d813937484a4ed26847947278d86cefe3ed0 not found: ID does not exist" Oct 30 19:58:34.536470 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.536437 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.538253 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.538231 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.540623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540596 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjk2w\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-kube-api-access-vjk2w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.540623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540625 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.540623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540642 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.541003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540660 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bb1deb-41a4-450c-a1c8-83ebd000e1ed-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.541003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540673 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.541003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540686 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6jt\" (UniqueName: \"kubernetes.io/projected/7239b6c0-a0b9-4040-b25d-ada66fd09937-kube-api-access-mx6jt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.541003 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.540701 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7239b6c0-a0b9-4040-b25d-ada66fd09937-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.557495 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.556412 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:58:34.563575 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.563548 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-mczrx"] Oct 30 19:58:34.619755 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.619717 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:58:34.626421 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.626324 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-s9zd5"] Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642071 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642138 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642322 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642381 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642457 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642529 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642731 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642776 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.643958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.642817 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\") pod \"854bf2a9-4868-4b62-a01e-71c85911e4ff\" (UID: \"854bf2a9-4868-4b62-a01e-71c85911e4ff\") " Oct 30 19:58:34.645688 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.645431 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.646666 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.646387 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.647392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.647344 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.647646 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.647628 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.647949 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.647894 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "kube-api-access-pr4fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.648043 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.647984 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.652274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.649996 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.652274 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.651692 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.652462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.652277 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo" pod "854bf2a9-4868-4b62-a01e-71c85911e4ff" (UID: "854bf2a9-4868-4b62-a01e-71c85911e4ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:34.689160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.689118 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" path="/var/lib/kubelet/pods/59bb1deb-41a4-450c-a1c8-83ebd000e1ed/volumes" Oct 30 19:58:34.690424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.690399 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" path="/var/lib/kubelet/pods/7239b6c0-a0b9-4040-b25d-ada66fd09937/volumes" Oct 30 19:58:34.747280 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747250 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747514 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747499 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854bf2a9-4868-4b62-a01e-71c85911e4ff-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747594 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747690 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747679 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747769 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747759 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747839 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747915 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4fj\" (UniqueName: \"kubernetes.io/projected/854bf2a9-4868-4b62-a01e-71c85911e4ff-kube-api-access-pr4fj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.747996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.747987 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.748072 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.748064 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854bf2a9-4868-4b62-a01e-71c85911e4ff-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.762046 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.762020 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:58:34.865948 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.865861 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:58:34.953099 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953067 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.953272 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953193 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.953324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953310 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.953473 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953450 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.953693 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953666 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.953797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953574 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.953797 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953702 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.954223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953860 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.954223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.953898 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.954223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.954011 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.954223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.954015 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.954223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.954068 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\") pod \"f71904f1-8891-4579-ba29-dd657af39ccb\" (UID: \"f71904f1-8891-4579-ba29-dd657af39ccb\") " Oct 30 19:58:34.954479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.954309 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.954479 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.954342 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.955051 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955025 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71904f1-8891-4579-ba29-dd657af39ccb-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.955143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955056 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.955143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955069 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.955143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955082 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.955143 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955096 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.956005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.955977 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:34.956567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.956542 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.957217 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.957198 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:34.957265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:34.957247 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk" pod "f71904f1-8891-4579-ba29-dd657af39ccb" (UID: "f71904f1-8891-4579-ba29-dd657af39ccb"). InnerVolumeSpecName "kube-api-access-t9svk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.056325 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056292 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056339 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056468 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056630 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.056706 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056677 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056761 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056746 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056772 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.056873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056853 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.056928 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056873 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:35.056928 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056900 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.057032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056954 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.057032 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.056971 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.058655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.057216 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\") pod \"97aa5c56-8fda-4878-9893-04074799918d\" (UID: \"97aa5c56-8fda-4878-9893-04074799918d\") " Oct 30 19:58:35.058655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.058617 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.058997 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.058978 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71904f1-8891-4579-ba29-dd657af39ccb-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059004 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71904f1-8891-4579-ba29-dd657af39ccb-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059017 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059029 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa5c56-8fda-4878-9893-04074799918d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059042 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059057 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059083 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059073 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9svk\" (UniqueName: \"kubernetes.io/projected/f71904f1-8891-4579-ba29-dd657af39ccb-kube-api-access-t9svk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059087 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059101 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.059926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.059902 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:35.060314 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.060291 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.061170 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.061150 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.061832 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.061811 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb" pod "97aa5c56-8fda-4878-9893-04074799918d" (UID: "97aa5c56-8fda-4878-9893-04074799918d"). InnerVolumeSpecName "kube-api-access-9zjxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.202888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.202188 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjxb\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-kube-api-access-9zjxb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.202888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.202228 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa5c56-8fda-4878-9893-04074799918d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.202888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.202243 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa5c56-8fda-4878-9893-04074799918d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.202888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.202255 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa5c56-8fda-4878-9893-04074799918d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.339004 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.338971 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.407804 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.407855 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.407990 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408051 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408154 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408193 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408249 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408344 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.408406 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\") pod \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\" (UID: \"f08adf74-3256-4c8e-8de3-d34a2b31be5c\") " Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.410089 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.413453 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.411099 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.414826 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.414469 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.417634 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.416124 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.417762 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.417707 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:35.450015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.438678 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.450015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.443475 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:35.450015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.443492 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "kube-api-access-brvjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.450015 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.449950 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy" pod "f08adf74-3256-4c8e-8de3-d34a2b31be5c" (UID: "f08adf74-3256-4c8e-8de3-d34a2b31be5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.452391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.451892 5991 generic.go:356] "Generic (PLEG): container finished" podID="97aa5c56-8fda-4878-9893-04074799918d" containerID="0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9" exitCode=0 Oct 30 19:58:35.452391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.451944 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" Oct 30 19:58:35.452391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.451994 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" event={"ID":"97aa5c56-8fda-4878-9893-04074799918d","Type":"ContainerDied","Data":"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9"} Oct 30 19:58:35.452391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.452046 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp" event={"ID":"97aa5c56-8fda-4878-9893-04074799918d","Type":"ContainerDied","Data":"9d9c61e6a493f1cbbc5082a6402206f87cb2d270d0928f9db62a0b49ac103e56"} Oct 30 19:58:35.452391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.452075 5991 scope.go:122] "RemoveContainer" containerID="0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9" Oct 30 19:58:35.457777 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.457718 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" event={"ID":"854bf2a9-4868-4b62-a01e-71c85911e4ff","Type":"ContainerDied","Data":"1d302e0a0610b97e41442fc9ce14fa0a8e7b7dd6256db243217fbed45c5f481c"} Oct 30 19:58:35.458162 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.458122 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl" Oct 30 19:58:35.460567 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.460355 5991 generic.go:356] "Generic (PLEG): container finished" podID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerID="f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218" exitCode=0 Oct 30 19:58:35.460688 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.460577 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" Oct 30 19:58:35.460901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.460799 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" event={"ID":"f08adf74-3256-4c8e-8de3-d34a2b31be5c","Type":"ContainerDied","Data":"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218"} Oct 30 19:58:35.460901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.460862 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx" event={"ID":"f08adf74-3256-4c8e-8de3-d34a2b31be5c","Type":"ContainerDied","Data":"ac9438dc44aa5f25a7c49b9b5fdfca207bfa65c26e993f8fc935930ffb683260"} Oct 30 19:58:35.466438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.462846 5991 generic.go:356] "Generic (PLEG): container finished" podID="f71904f1-8891-4579-ba29-dd657af39ccb" containerID="909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f" exitCode=0 Oct 30 19:58:35.466438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.462990 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" event={"ID":"f71904f1-8891-4579-ba29-dd657af39ccb","Type":"ContainerDied","Data":"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f"} Oct 30 19:58:35.466438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.463037 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" event={"ID":"f71904f1-8891-4579-ba29-dd657af39ccb","Type":"ContainerDied","Data":"294ec602cc6b35f9ebe8f8fafbf2ee1aa362fe6c981fb8e679bf3c49fd0f3e73"} Oct 30 19:58:35.466438 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.463114 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw" Oct 30 19:58:35.474757 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.474523 5991 scope.go:122] "RemoveContainer" containerID="0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9" Oct 30 19:58:35.474940 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:35.474920 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9\": container with ID starting with 0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9 not found: ID does not exist" containerID="0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9" Oct 30 19:58:35.475013 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.474954 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9"} err="failed to get container status \"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9\": rpc error: code = NotFound desc = could not find container \"0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9\": container with ID starting with 0ae6b97de885d3b36bc3ece5d6ac7c2e05a2495fb9c8b999b2c4b366186087f9 not found: ID does not exist" Oct 30 19:58:35.475013 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.474977 5991 scope.go:122] "RemoveContainer" containerID="a353b2118a782d482401de021b16555b1f4f9a6720e71a6e77bb12ee373ec0f6" Oct 30 19:58:35.487668 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.487638 5991 scope.go:122] "RemoveContainer" containerID="f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218" Oct 30 19:58:35.511097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511064 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511092 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511097 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511103 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511113 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511122 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvjb\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-kube-api-access-brvjb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511132 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511140 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511148 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.511504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.511156 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08adf74-3256-4c8e-8de3-d34a2b31be5c-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.512967 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.512945 5991 scope.go:122] "RemoveContainer" containerID="f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218" Oct 30 19:58:35.513288 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:35.513264 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218\": container with ID starting with f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218 not found: ID does not exist" containerID="f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218" Oct 30 19:58:35.513407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.513298 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218"} err="failed to get container status \"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218\": rpc error: code = NotFound desc = could not find container \"f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218\": container with ID starting with f016656b41d4924ddca44b5670f7f1ded5966362b43da79f237bc204ac4b4218 not found: ID does not exist" Oct 30 19:58:35.513407 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.513318 5991 scope.go:122] "RemoveContainer" containerID="909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f" Oct 30 19:58:35.524842 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.524812 5991 scope.go:122] "RemoveContainer" containerID="909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f" Oct 30 19:58:35.525177 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:35.525153 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f\": container with ID starting with 909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f not found: ID does not exist" containerID="909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f" Oct 30 19:58:35.525255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.525186 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f"} err="failed to get container status \"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f\": rpc error: code = NotFound desc = could not find container \"909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f\": container with ID starting with 909fa3fb950893418d0ba6178cd843bf126e6c4270116dfaac5bed6f461e2a8f not found: ID does not exist" Oct 30 19:58:35.558866 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.558828 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:58:35.564784 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.564753 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-479tx"] Oct 30 19:58:35.599649 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.599618 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:58:35.604103 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.604080 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-7svhp"] Oct 30 19:58:35.636838 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.636803 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:58:35.640530 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.640505 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-kkdhl"] Oct 30 19:58:35.676507 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.676473 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:58:35.680193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.680168 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-fxcqw"] Oct 30 19:58:35.750292 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.750252 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 19:58:35.751161 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751142 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" Oct 30 19:58:35.751214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751164 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" Oct 30 19:58:35.751214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751185 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" Oct 30 19:58:35.751214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751190 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" Oct 30 19:58:35.751214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751203 5991 cpu_manager.go:414] "containerMap: removing container" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" Oct 30 19:58:35.751214 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751208 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751246 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751254 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751264 5991 cpu_manager.go:414] "containerMap: removing container" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751269 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751282 5991 cpu_manager.go:414] "containerMap: removing container" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751287 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" Oct 30 19:58:35.751419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751415 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aa5c56-8fda-4878-9893-04074799918d" containerName="istio-proxy" Oct 30 19:58:35.751627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751431 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" containerName="istio-proxy" Oct 30 19:58:35.751627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751438 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" containerName="istio-proxy" Oct 30 19:58:35.751627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751445 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bb1deb-41a4-450c-a1c8-83ebd000e1ed" containerName="istio-proxy" Oct 30 19:58:35.751627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751453 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7239b6c0-a0b9-4040-b25d-ada66fd09937" containerName="istio-proxy" Oct 30 19:58:35.751627 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.751462 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" containerName="istio-proxy" Oct 30 19:58:35.757308 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.757286 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.772014 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.771990 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d96913-serving-cert\"" Oct 30 19:58:35.782903 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.782873 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 19:58:35.816898 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.816862 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.817047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.816915 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.817104 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.817039 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhjkw\" (UniqueName: \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.817188 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.817171 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.919312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.919272 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.919312 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.919321 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.919596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.919374 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhjkw\" (UniqueName: \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.919596 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.919476 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.919596 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:35.919481 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-serving-cert: secret "maas-api-e2e-api-b-d96913-serving-cert" not found Oct 30 19:58:35.919596 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:35.919546 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls podName:8d573997-3994-4a35-af27-bbc80c1b7c26 nodeName:}" failed. No retries permitted until 2026-10-30 19:58:36.419530475 +0000 UTC m=+1383.869023182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls") pod "maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" (UID: "8d573997-3994-4a35-af27-bbc80c1b7c26") : secret "maas-api-e2e-api-b-d96913-serving-cert" not found Oct 30 19:58:35.920113 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.920092 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.921888 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.921870 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:35.939520 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:35.939489 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhjkw\" (UniqueName: \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:36.429125 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.429030 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:36.431611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.431575 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") pod \"maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:36.672398 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.672344 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:36.682021 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.681959 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854bf2a9-4868-4b62-a01e-71c85911e4ff" path="/var/lib/kubelet/pods/854bf2a9-4868-4b62-a01e-71c85911e4ff/volumes" Oct 30 19:58:36.682550 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.682536 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97aa5c56-8fda-4878-9893-04074799918d" path="/var/lib/kubelet/pods/97aa5c56-8fda-4878-9893-04074799918d/volumes" Oct 30 19:58:36.683202 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.683182 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08adf74-3256-4c8e-8de3-d34a2b31be5c" path="/var/lib/kubelet/pods/f08adf74-3256-4c8e-8de3-d34a2b31be5c/volumes" Oct 30 19:58:36.684296 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.684277 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71904f1-8891-4579-ba29-dd657af39ccb" path="/var/lib/kubelet/pods/f71904f1-8891-4579-ba29-dd657af39ccb/volumes" Oct 30 19:58:36.999767 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.997540 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 19:58:36.999767 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:36.998667 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d573997_3994_4a35_af27_bbc80c1b7c26.slice/crio-d427843db961eb25befd30e035920f624751076f743fb2e547abf3ad4662cee5 WatchSource:0}: Error finding container d427843db961eb25befd30e035920f624751076f743fb2e547abf3ad4662cee5: Status 404 returned error can't find the container with id d427843db961eb25befd30e035920f624751076f743fb2e547abf3ad4662cee5 Oct 30 19:58:36.999901 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:36.999842 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:58:37.486595 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:37.486557 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" event={"ID":"8d573997-3994-4a35-af27-bbc80c1b7c26","Type":"ContainerStarted","Data":"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d"} Oct 30 19:58:37.487016 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:37.486620 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" event={"ID":"8d573997-3994-4a35-af27-bbc80c1b7c26","Type":"ContainerStarted","Data":"d427843db961eb25befd30e035920f624751076f743fb2e547abf3ad4662cee5"} Oct 30 19:58:37.487016 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:37.486650 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:37.530641 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:37.530581 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" podStartSLOduration=2.203210926 podStartE2EDuration="2.530564584s" totalImagesPullingTime="327.353658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:37.527721541 +0000 UTC m=+1384.977214265" watchObservedRunningTime="2026-10-30 19:58:37.530564584 +0000 UTC m=+1384.980057367" Oct 30 19:58:40.810142 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.810096 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 19:58:40.816380 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.816337 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.832265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.832233 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aa42c3-openshift-default-dockercfg-mrqh6\"" Oct 30 19:58:40.855704 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.855665 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 19:58:40.912539 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912496 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.912695 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912559 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.912756 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912706 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.912789 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912757 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.912886 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912865 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.912924 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.912910 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.913022 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.913001 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.913022 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.913023 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njg5h\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:40.913217 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:40.913136 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.016662 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.016738 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.016845 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.016906 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.016977 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.017029 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.017095 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.017123 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njg5h\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.017251 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.018010 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.018158 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.018435 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.018240 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.019057 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.018512 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.019317 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.019288 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.036416 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.034737 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.036416 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.035233 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.060439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.053583 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njg5h\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.087387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.087096 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\") pod \"e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.135387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.134475 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:41.608240 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:41.608203 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd6681a_76e7_4482_8f0c_4ee3386540ab.slice/crio-0bf2fc86eacd5c02cf47f1cdea96b96b3706f0d2ac83adca99d82ce90fdc59bf WatchSource:0}: Error finding container 0bf2fc86eacd5c02cf47f1cdea96b96b3706f0d2ac83adca99d82ce90fdc59bf: Status 404 returned error can't find the container with id 0bf2fc86eacd5c02cf47f1cdea96b96b3706f0d2ac83adca99d82ce90fdc59bf Oct 30 19:58:41.609563 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.609534 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 19:58:41.611523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.611490 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:41.611611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.611560 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:41.611611 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:41.611588 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:42.582173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:42.582121 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerStarted","Data":"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed"} Oct 30 19:58:42.582661 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:42.582241 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerStarted","Data":"0bf2fc86eacd5c02cf47f1cdea96b96b3706f0d2ac83adca99d82ce90fdc59bf"} Oct 30 19:58:42.634823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:42.634744 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podStartSLOduration=2.6347226089999998 podStartE2EDuration="2.634722609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:42.630467431 +0000 UTC m=+1390.079960154" watchObservedRunningTime="2026-10-30 19:58:42.634722609 +0000 UTC m=+1390.084215336" Oct 30 19:58:43.135807 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:43.135772 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:43.142301 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:43.142268 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:43.501868 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:43.501840 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 19:58:43.598615 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:43.598580 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:43.599746 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:43.599723 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 19:58:47.612415 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.612331 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 19:58:47.617811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.617783 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.635499 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.635471 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 19:58:47.650388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.643042 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aa42c3-dockercfg-fzhtr\"" Oct 30 19:58:47.650388 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.643551 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aa42c3\"" Oct 30 19:58:47.679482 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.679451 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 19:58:47.696064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.696016 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdtd\" (UniqueName: \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.696250 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.696228 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.805462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.800575 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.805462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.800950 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdtd\" (UniqueName: \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.805462 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.802529 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:47.816419 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.809682 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:47.911440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.900696 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 19:58:47.911440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.903612 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvsf7\" (UniqueName: \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:47.911440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.903766 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:47.969448 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:47.965267 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdtd\" (UniqueName: \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\") pod \"payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:48.007900 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.007867 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvsf7\" (UniqueName: \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:48.008102 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.007957 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:48.008969 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.008945 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:48.035978 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.035945 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvsf7\" (UniqueName: \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\") pod \"payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:48.242241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.242186 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:48.312735 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.312682 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:48.584768 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.584738 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 19:58:48.644904 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.644871 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" event={"ID":"a77ff384-212f-45ad-8b80-28a62b255b6f","Type":"ContainerStarted","Data":"d4adfa1fbb13d10472977cc63f8710bfe70ad6b45afad70f0e699dd7227813b1"} Oct 30 19:58:48.686789 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:48.686762 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3708da6_b791_440b_a283_df0be15c2257.slice/crio-95aa59bfc2b7419c7a4b15dd6e9ea312b737337af830232d8b9f7812c137703e WatchSource:0}: Error finding container 95aa59bfc2b7419c7a4b15dd6e9ea312b737337af830232d8b9f7812c137703e: Status 404 returned error can't find the container with id 95aa59bfc2b7419c7a4b15dd6e9ea312b737337af830232d8b9f7812c137703e Oct 30 19:58:48.689269 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:48.689228 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 19:58:49.657210 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.657164 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" event={"ID":"a77ff384-212f-45ad-8b80-28a62b255b6f","Type":"ContainerStarted","Data":"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf"} Oct 30 19:58:49.657660 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.657485 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:58:49.659737 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.659703 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" event={"ID":"c3708da6-b791-440b-a283-df0be15c2257","Type":"ContainerStarted","Data":"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061"} Oct 30 19:58:49.659873 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.659763 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" event={"ID":"c3708da6-b791-440b-a283-df0be15c2257","Type":"ContainerStarted","Data":"95aa59bfc2b7419c7a4b15dd6e9ea312b737337af830232d8b9f7812c137703e"} Oct 30 19:58:49.660548 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.660528 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:58:49.691743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.691707 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 19:58:49.698358 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.698329 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:49.702914 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.702853 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" podStartSLOduration=2.7028354390000002 podStartE2EDuration="2.702835439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:49.701888344 +0000 UTC m=+1397.151381070" watchObservedRunningTime="2026-10-30 19:58:49.702835439 +0000 UTC m=+1397.152328187" Oct 30 19:58:49.717778 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.717745 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-87ff9f\"" Oct 30 19:58:49.717941 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.717857 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-87ff9f-dockercfg-qhq4j\"" Oct 30 19:58:49.719064 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.719038 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 19:58:49.796821 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.795176 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" podStartSLOduration=2.795157685 podStartE2EDuration="2.795157685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:49.789934486 +0000 UTC m=+1397.239427212" watchObservedRunningTime="2026-10-30 19:58:49.795157685 +0000 UTC m=+1397.244650411" Oct 30 19:58:49.801239 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.801206 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 19:58:49.807124 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.807097 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:49.826955 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.826922 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 19:58:49.861080 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.861037 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28pv\" (UniqueName: \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:49.861283 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:49.861222 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.029391 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.003166 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9f9\" (UniqueName: \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.074877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.074017 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28pv\" (UniqueName: \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.074877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.074251 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.074877 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.074386 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.075734 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.075709 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.181932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.176415 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.181932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.176513 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9f9\" (UniqueName: \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.181932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.178513 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.181932 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.179233 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28pv\" (UniqueName: \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\") pod \"payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.198600 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.198568 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9f9\" (UniqueName: \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\") pod \"payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.336446 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.329761 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:50.438395 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.434144 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:50.677822 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:50.677789 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90b1645_a10d_475c_afa8_3c6c79c58cd2.slice/crio-8cabc27d7be36ed9a86d7e347068c568a3a89eccb040e6494ac314b749dd41d6 WatchSource:0}: Error finding container 8cabc27d7be36ed9a86d7e347068c568a3a89eccb040e6494ac314b749dd41d6: Status 404 returned error can't find the container with id 8cabc27d7be36ed9a86d7e347068c568a3a89eccb040e6494ac314b749dd41d6 Oct 30 19:58:50.683827 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.683804 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 19:58:50.779408 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:50.774567 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 19:58:50.779408 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:58:50.774849 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf684b144_bb7d_49fa_a3d7_fac34fb54598.slice/crio-bc23ef5d5cebb9024e0ce7e49aa6ef5ee9322f2107f9a4d92203490cc53d4555 WatchSource:0}: Error finding container bc23ef5d5cebb9024e0ce7e49aa6ef5ee9322f2107f9a4d92203490cc53d4555: Status 404 returned error can't find the container with id bc23ef5d5cebb9024e0ce7e49aa6ef5ee9322f2107f9a4d92203490cc53d4555 Oct 30 19:58:50.875882 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:58:50.875851 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:58:51.678763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.678717 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" event={"ID":"f684b144-bb7d-49fa-a3d7-fac34fb54598","Type":"ContainerStarted","Data":"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7"} Oct 30 19:58:51.679193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.678784 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" event={"ID":"f684b144-bb7d-49fa-a3d7-fac34fb54598","Type":"ContainerStarted","Data":"bc23ef5d5cebb9024e0ce7e49aa6ef5ee9322f2107f9a4d92203490cc53d4555"} Oct 30 19:58:51.679193 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.678816 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:58:51.680606 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.680579 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" event={"ID":"d90b1645-a10d-475c-afa8-3c6c79c58cd2","Type":"ContainerStarted","Data":"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3"} Oct 30 19:58:51.680753 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.680629 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" event={"ID":"d90b1645-a10d-475c-afa8-3c6c79c58cd2","Type":"ContainerStarted","Data":"8cabc27d7be36ed9a86d7e347068c568a3a89eccb040e6494ac314b749dd41d6"} Oct 30 19:58:51.680753 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.680722 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:58:51.721050 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.720996 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" podStartSLOduration=2.720980764 podStartE2EDuration="2.720980764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:51.719640447 +0000 UTC m=+1399.169133184" watchObservedRunningTime="2026-10-30 19:58:51.720980764 +0000 UTC m=+1399.170473530" Oct 30 19:58:51.763555 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:58:51.763488 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" podStartSLOduration=2.763474351 podStartE2EDuration="2.763474351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:51.76064303 +0000 UTC m=+1399.210135758" watchObservedRunningTime="2026-10-30 19:58:51.763474351 +0000 UTC m=+1399.212967076" Oct 30 19:59:00.681322 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:00.681288 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 19:59:01.684260 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.684228 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 19:59:01.902059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.902024 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 19:59:01.907284 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.907261 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:01.922259 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.922228 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-eb16ce\"" Oct 30 19:59:01.922481 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.922331 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-eb16ce-dockercfg-x76fg\"" Oct 30 19:59:01.925036 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.925010 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 19:59:01.926425 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.926355 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89xd\" (UniqueName: \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:01.926579 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:01.926558 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.093785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.047055 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89xd\" (UniqueName: \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.093785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.091422 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.093785 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.092348 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.192178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.192109 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 19:59:02.265387 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.265162 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89xd\" (UniqueName: \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\") pod \"payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.282779 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.282743 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 19:59:02.283478 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.283453 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.365760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.365672 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7b2\" (UniqueName: \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.365918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.365904 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.467422 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.467389 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.467620 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.467608 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7b2\" (UniqueName: \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.468130 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.468111 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.488163 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.488129 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7b2\" (UniqueName: \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\") pod \"payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.523963 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.523932 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:02.621505 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.620938 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:02.691156 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.691124 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 19:59:02.696434 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.696409 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 19:59:02.849936 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:59:02.849898 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04d33bc_e2e6_4aa5_938e_8661fe22a136.slice/crio-cf68faed11dcbf97cba4fa4aac465e7d1d8305907eacf31a37d725d15d7061d1 WatchSource:0}: Error finding container cf68faed11dcbf97cba4fa4aac465e7d1d8305907eacf31a37d725d15d7061d1: Status 404 returned error can't find the container with id cf68faed11dcbf97cba4fa4aac465e7d1d8305907eacf31a37d725d15d7061d1 Oct 30 19:59:02.851831 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.851784 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 19:59:02.961095 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:59:02.959729 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88763cf9_b783_4581_b1ed_fcaeaeb26a75.slice/crio-dc0648ca1c91ebd93c338227450a8b3ce7ae4a286a7d4b43abcc3be3d5d0e49c WatchSource:0}: Error finding container dc0648ca1c91ebd93c338227450a8b3ce7ae4a286a7d4b43abcc3be3d5d0e49c: Status 404 returned error can't find the container with id dc0648ca1c91ebd93c338227450a8b3ce7ae4a286a7d4b43abcc3be3d5d0e49c Oct 30 19:59:02.962257 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:02.962226 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 19:59:03.817047 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.817009 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" event={"ID":"88763cf9-b783-4581-b1ed-fcaeaeb26a75","Type":"ContainerStarted","Data":"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0"} Oct 30 19:59:03.817494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.817068 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" event={"ID":"88763cf9-b783-4581-b1ed-fcaeaeb26a75","Type":"ContainerStarted","Data":"dc0648ca1c91ebd93c338227450a8b3ce7ae4a286a7d4b43abcc3be3d5d0e49c"} Oct 30 19:59:03.817494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.817095 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:03.818812 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.818780 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" event={"ID":"c04d33bc-e2e6-4aa5-938e-8661fe22a136","Type":"ContainerStarted","Data":"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43"} Oct 30 19:59:03.818945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.818838 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:03.818945 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.818856 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" event={"ID":"c04d33bc-e2e6-4aa5-938e-8661fe22a136","Type":"ContainerStarted","Data":"cf68faed11dcbf97cba4fa4aac465e7d1d8305907eacf31a37d725d15d7061d1"} Oct 30 19:59:03.859935 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.859880 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" podStartSLOduration=2.859861547 podStartE2EDuration="2.859861547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.857517533 +0000 UTC m=+1411.307010261" watchObservedRunningTime="2026-10-30 19:59:03.859861547 +0000 UTC m=+1411.309354322" Oct 30 19:59:03.899131 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:03.899077 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" podStartSLOduration=2.89906411 podStartE2EDuration="2.89906411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.897740769 +0000 UTC m=+1411.347233495" watchObservedRunningTime="2026-10-30 19:59:03.89906411 +0000 UTC m=+1411.348556834" Oct 30 19:59:13.690289 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.690253 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:59:13.690813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.690556 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" containerID="cri-o://8ad4b9ed7116c4b088fa3aaa9e7a7d0be5a1e9cc217ca3398945c5aea10952b5" gracePeriod=30 Oct 30 19:59:13.702205 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.702166 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:59:13.702486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.702462 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" containerID="cri-o://21274d707dc533ba4378c8ba966edf2cf526578e3534afb24dd9bcb3d1a2974c" gracePeriod=30 Oct 30 19:59:13.707848 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.707814 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:59:13.708073 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:13.708051 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" containerID="cri-o://1eaf5a70dc3c17288925a9ff75fdea88e974f9744b309920ba4bd212830d340e" gracePeriod=30 Oct 30 19:59:14.835281 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:14.835253 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 19:59:14.835775 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:14.835317 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 19:59:18.967751 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.967340 5991 generic.go:356] "Generic (PLEG): container finished" podID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerID="8ad4b9ed7116c4b088fa3aaa9e7a7d0be5a1e9cc217ca3398945c5aea10952b5" exitCode=0 Oct 30 19:59:18.967751 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.967613 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" event={"ID":"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21","Type":"ContainerDied","Data":"8ad4b9ed7116c4b088fa3aaa9e7a7d0be5a1e9cc217ca3398945c5aea10952b5"} Oct 30 19:59:18.971469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.971340 5991 generic.go:356] "Generic (PLEG): container finished" podID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerID="21274d707dc533ba4378c8ba966edf2cf526578e3534afb24dd9bcb3d1a2974c" exitCode=0 Oct 30 19:59:18.971788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.971742 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" event={"ID":"5de6513d-f9e2-4166-9390-7e8ac17ef3f5","Type":"ContainerDied","Data":"21274d707dc533ba4378c8ba966edf2cf526578e3534afb24dd9bcb3d1a2974c"} Oct 30 19:59:18.975571 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.975211 5991 generic.go:356] "Generic (PLEG): container finished" podID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerID="1eaf5a70dc3c17288925a9ff75fdea88e974f9744b309920ba4bd212830d340e" exitCode=0 Oct 30 19:59:18.976384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:18.976113 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" event={"ID":"58fe2ac9-4d7f-405c-ac64-236c2166abfe","Type":"ContainerDied","Data":"1eaf5a70dc3c17288925a9ff75fdea88e974f9744b309920ba4bd212830d340e"} Oct 30 19:59:19.070504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.070465 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:59:19.073697 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.073661 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:59:19.135924 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.135893 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:59:19.211908 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.211867 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.211921 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.211953 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212007 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212066 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212227 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212155 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212227 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212217 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212293 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212248 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212324 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212299 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212356 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212329 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212431 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212410 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212508 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212489 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212563 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212530 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212615 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212578 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212670 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212637 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\") pod \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\" (UID: \"5de6513d-f9e2-4166-9390-7e8ac17ef3f5\") " Oct 30 19:59:19.212729 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212672 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212823 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212804 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.212879 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.212860 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\") pod \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\" (UID: \"58fe2ac9-4d7f-405c-ac64-236c2166abfe\") " Oct 30 19:59:19.214144 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.214110 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.214330 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.214311 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.215264 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.215231 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.215604 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.215562 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.215895 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.215871 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "kube-api-access-5mj64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.216197 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.216174 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:19.216335 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.216289 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.216641 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.216616 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.216992 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.216967 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:19.217276 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.217254 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.217468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.217448 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.218886 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.218822 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.219669 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.219637 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.220718 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.220694 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:19.220817 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.220754 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.220958 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.220943 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:19.221800 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.221779 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy" pod "5de6513d-f9e2-4166-9390-7e8ac17ef3f5" (UID: "5de6513d-f9e2-4166-9390-7e8ac17ef3f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.222455 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.222426 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp" pod "58fe2ac9-4d7f-405c-ac64-236c2166abfe" (UID: "58fe2ac9-4d7f-405c-ac64-236c2166abfe"). InnerVolumeSpecName "kube-api-access-fl8wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.315187 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315146 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315257 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315457 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315416 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315466 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315509 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315538 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.315743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315609 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.315743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315654 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315658 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.315743 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315725 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315792 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.315939 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315845 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\") pod \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\" (UID: \"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21\") " Oct 30 19:59:19.316036 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.315983 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.316343 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316274 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:19.316788 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316762 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316794 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316807 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8wp\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-kube-api-access-fl8wp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316822 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316836 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mj64\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-kube-api-access-5mj64\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316848 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316863 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316877 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.316893 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316889 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316901 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316913 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316926 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316940 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316952 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316966 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316977 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.316990 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317004 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317017 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317031 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317043 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317055 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de6513d-f9e2-4166-9390-7e8ac17ef3f5-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.317326 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317068 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58fe2ac9-4d7f-405c-ac64-236c2166abfe-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.318157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.317835 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:19.318157 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.318115 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:19.318266 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.318155 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.318481 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.318455 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n" pod "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" (UID: "8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21"). InnerVolumeSpecName "kube-api-access-h4k7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:19.419100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.419061 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4k7n\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-kube-api-access-h4k7n\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.419100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.419099 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.419376 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.419113 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.419376 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.419129 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:19.985594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.985489 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" Oct 30 19:59:19.985594 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.985527 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd" event={"ID":"8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21","Type":"ContainerDied","Data":"b19e1a6934dab7f88603447f04c21f83235563d041e48ec2121e2adccb2b9615"} Oct 30 19:59:19.986220 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.985596 5991 scope.go:122] "RemoveContainer" containerID="8ad4b9ed7116c4b088fa3aaa9e7a7d0be5a1e9cc217ca3398945c5aea10952b5" Oct 30 19:59:19.987996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.987564 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" Oct 30 19:59:19.987996 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.987856 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k" event={"ID":"5de6513d-f9e2-4166-9390-7e8ac17ef3f5","Type":"ContainerDied","Data":"b519a9fb4e19ea38e91eddcf4db01ee5076c3b9885522acaf460fe286eaba9f2"} Oct 30 19:59:19.995265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.995227 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" Oct 30 19:59:19.995265 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.995240 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt" event={"ID":"58fe2ac9-4d7f-405c-ac64-236c2166abfe","Type":"ContainerDied","Data":"e3ba6061a3ec40b1e8c9f91074249ed7e316acc527dc35b3755c3f4f437d8ef1"} Oct 30 19:59:19.999504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:19.999481 5991 scope.go:122] "RemoveContainer" containerID="21274d707dc533ba4378c8ba966edf2cf526578e3534afb24dd9bcb3d1a2974c" Oct 30 19:59:20.012754 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.012724 5991 scope.go:122] "RemoveContainer" containerID="1eaf5a70dc3c17288925a9ff75fdea88e974f9744b309920ba4bd212830d340e" Oct 30 19:59:20.054413 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.054373 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:59:20.058809 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.058773 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-7zmkd"] Oct 30 19:59:20.102329 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.102295 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:59:20.107606 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.107102 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-t4hbt"] Oct 30 19:59:20.143541 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.143499 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:59:20.149063 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.149033 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-x4r7k"] Oct 30 19:59:20.682061 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.682026 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" path="/var/lib/kubelet/pods/58fe2ac9-4d7f-405c-ac64-236c2166abfe/volumes" Oct 30 19:59:20.682665 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.682643 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" path="/var/lib/kubelet/pods/5de6513d-f9e2-4166-9390-7e8ac17ef3f5/volumes" Oct 30 19:59:20.683160 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:20.683146 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" path="/var/lib/kubelet/pods/8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21/volumes" Oct 30 19:59:20.873374 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:20.873318 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:59:21.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:21.220613 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 19:59:25.022390 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.021072 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.262223 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.262174 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.477178 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.475226 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.477963 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.477926 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 19:59:25.478467 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.478440 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 19:59:25.568001 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.567952 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.570717 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.570677 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:59:25.628523 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.628467 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.630163 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.630122 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.673572 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:25.673516 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:26.038386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.029523 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:26.038386 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.030382 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:59:26.452267 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.452174 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 19:59:26.522384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.521481 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:26.522384 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.521918 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:59:26.522626 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.522555 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:59:26.525644 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.522625 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:59:26.560759 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.560724 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 19:59:26.561655 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:26.561633 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 19:59:27.396553 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.396510 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 19:59:27.397358 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397343 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" Oct 30 19:59:27.397424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397359 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" Oct 30 19:59:27.397424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397416 5991 cpu_manager.go:414] "containerMap: removing container" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" Oct 30 19:59:27.397424 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397421 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" Oct 30 19:59:27.397524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397454 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" Oct 30 19:59:27.397524 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397460 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" Oct 30 19:59:27.397590 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397586 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fe2ac9-4d7f-405c-ac64-236c2166abfe" containerName="istio-proxy" Oct 30 19:59:27.397623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397600 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4efdfa-8bdd-44c5-83cf-d9eac51f9e21" containerName="istio-proxy" Oct 30 19:59:27.397623 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.397608 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de6513d-f9e2-4166-9390-7e8ac17ef3f5" containerName="istio-proxy" Oct 30 19:59:27.402181 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.402150 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.430008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.429562 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 19:59:27.430008 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.429852 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aa42c3-serving-cert\"" Oct 30 19:59:27.493200 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.493158 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.493360 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.493232 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.493486 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.493447 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.493590 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.493571 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glfb\" (UniqueName: \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.595041 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.595000 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.595255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.595103 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glfb\" (UniqueName: \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.595255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.595176 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.595255 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.595222 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.596117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.596086 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.599134 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.599110 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.606590 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.606557 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.619917 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.619879 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glfb\" (UniqueName: \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\") pod \"maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:27.720445 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:27.720333 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:28.093241 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:28.092578 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 19:59:28.094829 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:59:28.094801 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349e784b_9359_45ec_92b2_cc7648e017a7.slice/crio-6b5a8e7986b74474722056082ad247bc58ce9c46e387a279647bbc3d27f0dc10 WatchSource:0}: Error finding container 6b5a8e7986b74474722056082ad247bc58ce9c46e387a279647bbc3d27f0dc10: Status 404 returned error can't find the container with id 6b5a8e7986b74474722056082ad247bc58ce9c46e387a279647bbc3d27f0dc10 Oct 30 19:59:28.516221 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:28.516183 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" event={"ID":"349e784b-9359-45ec-92b2-cc7648e017a7","Type":"ContainerStarted","Data":"6b5a8e7986b74474722056082ad247bc58ce9c46e387a279647bbc3d27f0dc10"} Oct 30 19:59:29.527578 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:29.527491 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" event={"ID":"349e784b-9359-45ec-92b2-cc7648e017a7","Type":"ContainerStarted","Data":"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2"} Oct 30 19:59:29.543173 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:29.529006 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:29.600714 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:29.576857 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" podStartSLOduration=2.257581847 podStartE2EDuration="2.576824999s" totalImagesPullingTime="319.243152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:29.570268369 +0000 UTC m=+1437.019761096" watchObservedRunningTime="2026-10-30 19:59:29.576824999 +0000 UTC m=+1437.026317727" Oct 30 19:59:31.063990 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.063948 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 19:59:31.069081 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.069055 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.095169 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.095124 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.095382 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.095226 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.095468 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.095356 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnn8\" (UniqueName: \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.095760 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.095735 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.099506 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.099475 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-87ff9f-serving-cert\"" Oct 30 19:59:31.106966 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.106930 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 19:59:31.197086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.197046 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.197282 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.197130 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.197282 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.197237 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnn8\" (UniqueName: \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.197597 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.197573 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.197764 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:31.197746 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-serving-cert: secret "maas-api-e2e-shared-b-w2-87ff9f-serving-cert" not found Oct 30 19:59:31.197879 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:31.197831 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls podName:10d5e21e-b639-44fd-b15f-fb410753e91e nodeName:}" failed. No retries permitted until 2026-10-30 19:59:31.697809747 +0000 UTC m=+1439.147302450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" (UID: "10d5e21e-b639-44fd-b15f-fb410753e91e") : secret "maas-api-e2e-shared-b-w2-87ff9f-serving-cert" not found Oct 30 19:59:31.197980 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.197962 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.200469 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.200439 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.219112 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.219078 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnn8\" (UniqueName: \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.713440 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.710499 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.730130 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.730089 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:31.990568 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:31.990207 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:32.318433 ip-10-0-132-4 kubenswrapper[5991]: W1030 19:59:32.318399 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d5e21e_b639_44fd_b15f_fb410753e91e.slice/crio-4d4f3dbfc9e7430996c7af6dd1ba1779124e1a884b98f8e077a87a129effe74e WatchSource:0}: Error finding container 4d4f3dbfc9e7430996c7af6dd1ba1779124e1a884b98f8e077a87a129effe74e: Status 404 returned error can't find the container with id 4d4f3dbfc9e7430996c7af6dd1ba1779124e1a884b98f8e077a87a129effe74e Oct 30 19:59:32.318918 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:32.318883 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 19:59:32.581346 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:32.581261 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" event={"ID":"10d5e21e-b639-44fd-b15f-fb410753e91e","Type":"ContainerStarted","Data":"4d4f3dbfc9e7430996c7af6dd1ba1779124e1a884b98f8e077a87a129effe74e"} Oct 30 19:59:33.594515 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:33.594474 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" event={"ID":"10d5e21e-b639-44fd-b15f-fb410753e91e","Type":"ContainerStarted","Data":"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4"} Oct 30 19:59:33.595028 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:33.594576 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:33.645516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:33.645468 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" podStartSLOduration=2.2830452 podStartE2EDuration="2.645452495s" totalImagesPullingTime="362.407295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:33.642987638 +0000 UTC m=+1441.092480365" watchObservedRunningTime="2026-10-30 19:59:33.645452495 +0000 UTC m=+1441.094945219" Oct 30 19:59:34.605589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.605490 5991 generic.go:356] "Generic (PLEG): container finished" podID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerID="125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1" exitCode=0 Oct 30 19:59:34.605589 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.605568 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerDied","Data":"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1"} Oct 30 19:59:34.606209 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.606192 5991 scope.go:122] "RemoveContainer" containerID="125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1" Oct 30 19:59:34.608211 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.608175 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:34.608389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.608261 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:34.608389 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:34.608295 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:35.615037 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:35.614994 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerStarted","Data":"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4"} Oct 30 19:59:35.996485 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:35.996451 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:59:35.997683 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:35.997658 5991 patch_prober.go:29] interesting pod/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:59:35.997813 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:35.997708 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:59:36.202694 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:36.202651 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:59:36.583674 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:36.583640 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 19:59:36.996817 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:36.996783 5991 patch_prober.go:29] interesting pod/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:59:36.997504 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:36.997469 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:59:37.997098 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:37.997057 5991 patch_prober.go:29] interesting pod/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:59:37.997763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:37.997728 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:59:38.996987 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:38.996944 5991 patch_prober.go:29] interesting pod/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:59:38.997196 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:38.997018 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:59:39.612309 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:39.612281 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 19:59:39.997490 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:39.997450 5991 patch_prober.go:29] interesting pod/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:59:39.997923 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:39.997507 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:59:40.115494 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:40.115448 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:59:41.203338 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.203282 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" podUID="021986b0-ee75-44ea-90b7-96752a2133de" containerName="payload-processing" containerID="cri-o://28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254" gracePeriod=25 Oct 30 19:59:41.221895 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.221848 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:59:41.224713 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.224682 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 19:59:41.527645 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.527616 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:59:41.559086 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.559055 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\" (UniqueName: \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\") pod \"021986b0-ee75-44ea-90b7-96752a2133de\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " Oct 30 19:59:41.559270 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.559184 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\") pod \"021986b0-ee75-44ea-90b7-96752a2133de\" (UID: \"021986b0-ee75-44ea-90b7-96752a2133de\") " Oct 30 19:59:41.559637 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.559604 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume" pod "021986b0-ee75-44ea-90b7-96752a2133de" (UID: "021986b0-ee75-44ea-90b7-96752a2133de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:41.559839 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.559822 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021986b0-ee75-44ea-90b7-96752a2133de-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:41.561477 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.561450 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn" pod "021986b0-ee75-44ea-90b7-96752a2133de" (UID: "021986b0-ee75-44ea-90b7-96752a2133de"). InnerVolumeSpecName "kube-api-access-kvkhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:41.661998 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.661958 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvkhn\" (UniqueName: \"kubernetes.io/projected/021986b0-ee75-44ea-90b7-96752a2133de-kube-api-access-kvkhn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:41.693727 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.693685 5991 generic.go:356] "Generic (PLEG): container finished" podID="021986b0-ee75-44ea-90b7-96752a2133de" containerID="28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254" exitCode=0 Oct 30 19:59:41.693926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.693748 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" Oct 30 19:59:41.693926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.693763 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" event={"ID":"021986b0-ee75-44ea-90b7-96752a2133de","Type":"ContainerDied","Data":"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254"} Oct 30 19:59:41.693926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.693819 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn" event={"ID":"021986b0-ee75-44ea-90b7-96752a2133de","Type":"ContainerDied","Data":"d271e3502ac18bcf806e26e1d03273c264c710c818b149166c68a24c75a9cdc1"} Oct 30 19:59:41.693926 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.693846 5991 scope.go:122] "RemoveContainer" containerID="28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254" Oct 30 19:59:41.706710 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.706687 5991 scope.go:122] "RemoveContainer" containerID="28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254" Oct 30 19:59:41.707047 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:41.707016 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254\": container with ID starting with 28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254 not found: ID does not exist" containerID="28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254" Oct 30 19:59:41.707155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.707056 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254"} err="failed to get container status \"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254\": rpc error: code = NotFound desc = could not find container \"28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254\": container with ID starting with 28f114a697673941b23de30be7d80b06535b1842faca17d46308b9637e0bb254 not found: ID does not exist" Oct 30 19:59:41.735535 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.735498 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 19:59:41.736439 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.736421 5991 cpu_manager.go:414] "containerMap: removing container" podUID="021986b0-ee75-44ea-90b7-96752a2133de" containerName="payload-processing" Oct 30 19:59:41.736582 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.736442 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021986b0-ee75-44ea-90b7-96752a2133de" containerName="payload-processing" Oct 30 19:59:41.736664 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.736653 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="021986b0-ee75-44ea-90b7-96752a2133de" containerName="payload-processing" Oct 30 19:59:41.740238 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.740184 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.754985 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.754955 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:59:41.759409 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.759325 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d97a5-5dfcc5fdbb-6xtkn"] Oct 30 19:59:41.763208 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.763172 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.763385 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.763306 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.764100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.763629 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.764100 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.763995 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jxm\" (UniqueName: \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.764923 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.764901 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 19:59:41.810763 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.810728 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eb16ce-serving-cert\"" Oct 30 19:59:41.865117 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.865079 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jxm\" (UniqueName: \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.865310 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.865134 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.865310 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.865250 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.865500 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:41.865460 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-serving-cert: secret "maas-api-e2e-inf-b-eb16ce-serving-cert" not found Oct 30 19:59:41.865555 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.865524 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.865636 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:41.865610 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls podName:a90bf0d9-749f-4791-b7b9-235f10512642 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:42.365586215 +0000 UTC m=+1449.815078934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls") pod "maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" (UID: "a90bf0d9-749f-4791-b7b9-235f10512642") : secret "maas-api-e2e-inf-b-eb16ce-serving-cert" not found Oct 30 19:59:41.866399 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.866344 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.867764 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.867747 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:41.886005 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:41.885975 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jxm\" (UniqueName: \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:42.376011 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:42.375968 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:42.378597 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:42.378561 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:42.657219 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:42.657130 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:42.702134 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:42.702094 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021986b0-ee75-44ea-90b7-96752a2133de" path="/var/lib/kubelet/pods/021986b0-ee75-44ea-90b7-96752a2133de/volumes" Oct 30 19:59:43.001029 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.000117 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 19:59:43.723548 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.723501 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" event={"ID":"a90bf0d9-749f-4791-b7b9-235f10512642","Type":"ContainerStarted","Data":"67cd7072da1853404c5bc458a49691bebc1ea9baba7f6cdd1cd6a675527f884f"} Oct 30 19:59:43.724162 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.724129 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" event={"ID":"a90bf0d9-749f-4791-b7b9-235f10512642","Type":"ContainerStarted","Data":"eae491cb04600b11ffc171dc0f6b26913d3b288445eebe789edc9c23cad980cf"} Oct 30 19:59:43.768084 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.768029 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" podStartSLOduration=2.394408045 podStartE2EDuration="2.768015049s" totalImagesPullingTime="373.607004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:43.766659894 +0000 UTC m=+1451.216152619" watchObservedRunningTime="2026-10-30 19:59:43.768015049 +0000 UTC m=+1451.217507819" Oct 30 19:59:43.991807 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.991722 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:59:43.992074 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:43.992045 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" containerID="cri-o://286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01" gracePeriod=30 Oct 30 19:59:44.731937 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:44.731907 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:49.347741 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.347712 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.437977 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438084 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438181 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438336 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438413 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438579 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438611 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438662 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.438758 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\") pod \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\" (UID: \"0e07c839-ca46-4f3e-9301-0d86384e4dc3\") " Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.439738 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:49.440392 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.440044 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:49.441122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.440915 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:49.441122 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.440991 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:49.451452 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.441963 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:49.465059 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.465012 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "kube-api-access-mpbgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:49.494516 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.494474 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:49.494699 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.494525 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:49.494699 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.494564 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token" pod "0e07c839-ca46-4f3e-9301-0d86384e4dc3" (UID: "0e07c839-ca46-4f3e-9301-0d86384e4dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:49.541234 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541191 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541234 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541233 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541247 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541260 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541272 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541285 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541297 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e07c839-ca46-4f3e-9301-0d86384e4dc3-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541312 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbgg\" (UniqueName: \"kubernetes.io/projected/0e07c839-ca46-4f3e-9301-0d86384e4dc3-kube-api-access-mpbgg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.541493 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.541327 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e07c839-ca46-4f3e-9301-0d86384e4dc3-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 19:59:49.776965 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.776931 5991 generic.go:356] "Generic (PLEG): container finished" podID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerID="286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01" exitCode=0 Oct 30 19:59:49.777155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.777005 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" Oct 30 19:59:49.777155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.777015 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" event={"ID":"0e07c839-ca46-4f3e-9301-0d86384e4dc3","Type":"ContainerDied","Data":"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01"} Oct 30 19:59:49.777155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.777079 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt" event={"ID":"0e07c839-ca46-4f3e-9301-0d86384e4dc3","Type":"ContainerDied","Data":"f312ab9e0d0134fe14ce251e5f540e77ea6474d6c05797388ac936ec4fecbd32"} Oct 30 19:59:49.777155 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.777109 5991 scope.go:122] "RemoveContainer" containerID="286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01" Oct 30 19:59:49.790002 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.789975 5991 scope.go:122] "RemoveContainer" containerID="286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01" Oct 30 19:59:49.790296 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:49.790274 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01\": container with ID starting with 286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01 not found: ID does not exist" containerID="286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01" Oct 30 19:59:49.790352 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.790306 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01"} err="failed to get container status \"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01\": rpc error: code = NotFound desc = could not find container \"286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01\": container with ID starting with 286f3980e78b2bd5d10e07534789fa7106af06a6d16b8829c18c196379b28c01 not found: ID does not exist" Oct 30 19:59:49.832144 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.832108 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:59:49.845303 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:49.845268 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-44mjt"] Oct 30 19:59:50.680811 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:50.680777 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" path="/var/lib/kubelet/pods/0e07c839-ca46-4f3e-9301-0d86384e4dc3/volumes" Oct 30 19:59:50.746218 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:50.746187 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 19:59:50.876827 ip-10-0-132-4 kubenswrapper[5991]: E1030 19:59:50.876785 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 19:59:59.640572 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:59.640526 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:59:59.641138 ip-10-0-132-4 kubenswrapper[5991]: I1030 19:59:59.640634 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:00.273397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.273337 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:00.274724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.274634 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" Oct 30 20:00:00.274724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.274658 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" Oct 30 20:00:00.275020 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.274933 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e07c839-ca46-4f3e-9301-0d86384e4dc3" containerName="istio-proxy" Oct 30 20:00:00.281749 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.281714 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.295388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.295341 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:00.338888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.338593 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:00.338888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.338719 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.361814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.361299 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:00.370304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.368595 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:00.370304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.368717 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.375456 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.373657 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:00.387089 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.387026 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:00:00.466751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.466709 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kv5r\" (UniqueName: \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.466940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.466894 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.467024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.466936 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhvx\" (UniqueName: \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.467087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.467030 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhv2m\" (UniqueName: \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.467087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.467069 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.467199 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.467100 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.569070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.568982 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.569070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569051 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhvx\" (UniqueName: \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.569315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569112 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhv2m\" (UniqueName: \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.569315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569156 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.569315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569197 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.569987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569376 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kv5r\" (UniqueName: \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.569987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.569871 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.570245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.570219 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.570637 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.570616 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.595173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.595143 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhvx\" (UniqueName: \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:00.595507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.595482 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kv5r\" (UniqueName: \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\") pod \"maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.597213 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.597178 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhv2m\" (UniqueName: \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\") pod \"maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.654328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.654283 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:00.681451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.681417 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:00.696557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:00.696474 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:01.003955 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:01.003923 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0a3fbf_ec23_43d7_a7cc_f052bf604305.slice/crio-a4a5a452188b85cc953d673e2fb5990f06ca0011de484f3ee3c60b8b222fbcc1 WatchSource:0}: Error finding container a4a5a452188b85cc953d673e2fb5990f06ca0011de484f3ee3c60b8b222fbcc1: Status 404 returned error can't find the container with id a4a5a452188b85cc953d673e2fb5990f06ca0011de484f3ee3c60b8b222fbcc1 Oct 30 20:00:01.004742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.004717 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:01.042739 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:01.042707 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff2c283_515f_410b_a98b_f59319bf2dd1.slice/crio-884a07a80c9cb6321946269fa8b1f86fb44b495981ee7736cfcabf4063885a9c WatchSource:0}: Error finding container 884a07a80c9cb6321946269fa8b1f86fb44b495981ee7736cfcabf4063885a9c: Status 404 returned error can't find the container with id 884a07a80c9cb6321946269fa8b1f86fb44b495981ee7736cfcabf4063885a9c Oct 30 20:00:01.043572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.043545 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:01.054578 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:01.054542 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2850e2dc_29b5_4e39_9973_19df757cdc06.slice/crio-a39c49acd70593bebd402c1ff82538ea48c7b3fa68647111280be7480baa1b43 WatchSource:0}: Error finding container a39c49acd70593bebd402c1ff82538ea48c7b3fa68647111280be7480baa1b43: Status 404 returned error can't find the container with id a39c49acd70593bebd402c1ff82538ea48c7b3fa68647111280be7480baa1b43 Oct 30 20:00:01.055129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.055106 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:01.900679 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.900640 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerStarted","Data":"a4a5a452188b85cc953d673e2fb5990f06ca0011de484f3ee3c60b8b222fbcc1"} Oct 30 20:00:01.901963 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.901934 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerStarted","Data":"a39c49acd70593bebd402c1ff82538ea48c7b3fa68647111280be7480baa1b43"} Oct 30 20:00:01.903634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.903610 5991 generic.go:356] "Generic (PLEG): container finished" podID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerID="00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed" exitCode=0 Oct 30 20:00:01.903752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.903680 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerDied","Data":"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed"} Oct 30 20:00:01.904245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.904228 5991 scope.go:122] "RemoveContainer" containerID="00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed" Oct 30 20:00:01.905263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.905004 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerStarted","Data":"884a07a80c9cb6321946269fa8b1f86fb44b495981ee7736cfcabf4063885a9c"} Oct 30 20:00:01.906178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.906149 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:01.906288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.906236 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:01.906402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:01.906288 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:02.919936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:02.919898 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerStarted","Data":"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f"} Oct 30 20:00:03.137033 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:03.136983 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:00:03.137925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:03.137604 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:03.137925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:03.137672 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:04.136472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:04.135997 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:04.136472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:04.136062 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:05.135583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:05.135542 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.135774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:05.135609 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:05.994865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:05.994806 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerStarted","Data":"ba79af6eba2e72185360c3a84c7e129cc2ef8df35f699f3c716a32f467dd1667"} Oct 30 20:00:06.037238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:06.037184 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" podStartSLOduration=1.7857648099999999 podStartE2EDuration="6.03716958s" totalImagesPullingTime="4.25140477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:06.035378529 +0000 UTC m=+1473.484871248" watchObservedRunningTime="2026-10-30 20:00:06.03716958 +0000 UTC m=+1473.486662304" Oct 30 20:00:06.135832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:06.135750 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.135832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:06.135802 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:07.005624 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.005576 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerStarted","Data":"c291fa8c0f67877e9265798fe13cc2795153895c07440542d0850e8755e5b213"} Oct 30 20:00:07.007509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.007478 5991 generic.go:356] "Generic (PLEG): container finished" podID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerID="ba79af6eba2e72185360c3a84c7e129cc2ef8df35f699f3c716a32f467dd1667" exitCode=60 Oct 30 20:00:07.007669 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.007533 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerDied","Data":"ba79af6eba2e72185360c3a84c7e129cc2ef8df35f699f3c716a32f467dd1667"} Oct 30 20:00:07.007895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.007863 5991 scope.go:122] "RemoveContainer" containerID="ba79af6eba2e72185360c3a84c7e129cc2ef8df35f699f3c716a32f467dd1667" Oct 30 20:00:07.009766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.009739 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerID="779167542de2fd996dc8765b7af7103080e984fbd96ac26b71256ef84fae78cd" exitCode=60 Oct 30 20:00:07.009863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.009777 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerDied","Data":"779167542de2fd996dc8765b7af7103080e984fbd96ac26b71256ef84fae78cd"} Oct 30 20:00:07.010286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.010270 5991 scope.go:122] "RemoveContainer" containerID="779167542de2fd996dc8765b7af7103080e984fbd96ac26b71256ef84fae78cd" Oct 30 20:00:07.056293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.056241 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" podStartSLOduration=2.140193445 podStartE2EDuration="7.056224596s" totalImagesPullingTime="4.916031151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.054882033 +0000 UTC m=+1474.504374758" watchObservedRunningTime="2026-10-30 20:00:07.056224596 +0000 UTC m=+1474.505717321" Oct 30 20:00:07.136040 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.136008 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.136218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:07.136063 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:08.019949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.019909 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerStarted","Data":"f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6"} Oct 30 20:00:08.021730 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.021695 5991 generic.go:356] "Generic (PLEG): container finished" podID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerID="c291fa8c0f67877e9265798fe13cc2795153895c07440542d0850e8755e5b213" exitCode=60 Oct 30 20:00:08.021875 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.021754 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerDied","Data":"c291fa8c0f67877e9265798fe13cc2795153895c07440542d0850e8755e5b213"} Oct 30 20:00:08.022144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.022126 5991 scope.go:122] "RemoveContainer" containerID="c291fa8c0f67877e9265798fe13cc2795153895c07440542d0850e8755e5b213" Oct 30 20:00:08.024178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.024154 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerStarted","Data":"04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b"} Oct 30 20:00:08.091185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.091110 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" podStartSLOduration=3.751725401 podStartE2EDuration="8.091086519s" totalImagesPullingTime="4.339361118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:08.081817699 +0000 UTC m=+1475.531310424" watchObservedRunningTime="2026-10-30 20:00:08.091086519 +0000 UTC m=+1475.540579241" Oct 30 20:00:08.138872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.138823 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.139096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:08.138900 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 20:00:09.035984 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.035934 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerID="f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6" exitCode=60 Oct 30 20:00:09.036556 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.036012 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerDied","Data":"f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6"} Oct 30 20:00:09.036556 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.036076 5991 scope.go:122] "RemoveContainer" containerID="779167542de2fd996dc8765b7af7103080e984fbd96ac26b71256ef84fae78cd" Oct 30 20:00:09.036556 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.036477 5991 scope.go:122] "RemoveContainer" containerID="f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6" Oct 30 20:00:09.036946 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:09.036920 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g_odh-ai-gateway-infra(3c0a3fbf-ec23-43d7-a7cc-f052bf604305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" Oct 30 20:00:09.038584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.038560 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerStarted","Data":"5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7"} Oct 30 20:00:09.040658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.040636 5991 generic.go:356] "Generic (PLEG): container finished" podID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerID="04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b" exitCode=60 Oct 30 20:00:09.040759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.040707 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerDied","Data":"04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b"} Oct 30 20:00:09.040957 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.040943 5991 scope.go:122] "RemoveContainer" containerID="04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b" Oct 30 20:00:09.041289 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:09.041270 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p_odh-ai-gateway-infra(1ff2c283-515f-410b-a98b-f59319bf2dd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" Oct 30 20:00:09.048936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.048915 5991 scope.go:122] "RemoveContainer" containerID="ba79af6eba2e72185360c3a84c7e129cc2ef8df35f699f3c716a32f467dd1667" Oct 30 20:00:09.142036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.142000 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:00:09.142518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.142495 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:00:09.143413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:09.143355 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:00:10.050843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.050807 5991 scope.go:122] "RemoveContainer" containerID="f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6" Oct 30 20:00:10.051289 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:10.051257 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g_odh-ai-gateway-infra(3c0a3fbf-ec23-43d7-a7cc-f052bf604305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" Oct 30 20:00:10.052631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.052605 5991 generic.go:356] "Generic (PLEG): container finished" podID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerID="5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7" exitCode=60 Oct 30 20:00:10.052756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.052689 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerDied","Data":"5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7"} Oct 30 20:00:10.052756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.052752 5991 scope.go:122] "RemoveContainer" containerID="c291fa8c0f67877e9265798fe13cc2795153895c07440542d0850e8755e5b213" Oct 30 20:00:10.053244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.053174 5991 scope.go:122] "RemoveContainer" containerID="5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7" Oct 30 20:00:10.053684 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:10.053657 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752_odh-ai-gateway-infra(2850e2dc-29b5-4e39-9973-19df757cdc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" Oct 30 20:00:10.055062 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:10.055043 5991 scope.go:122] "RemoveContainer" containerID="04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b" Oct 30 20:00:10.055451 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:10.055431 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p_odh-ai-gateway-infra(1ff2c283-515f-410b-a98b-f59319bf2dd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" Oct 30 20:00:11.064807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:11.064780 5991 scope.go:122] "RemoveContainer" containerID="5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7" Oct 30 20:00:11.065190 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:11.065167 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752_odh-ai-gateway-infra(2850e2dc-29b5-4e39-9973-19df757cdc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" Oct 30 20:00:19.215382 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.215317 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:00:19.222965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.222932 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.241652 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.241620 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:00:19.247662 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.247620 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-652137\"" Oct 30 20:00:19.247662 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.247633 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-652137-dockercfg-9hvf5\"" Oct 30 20:00:19.290401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.274615 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:00:19.348385 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.347049 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.348385 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.347417 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gv4\" (UniqueName: \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.437401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.436606 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:00:19.437401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.436745 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.476602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.463556 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.476602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.463743 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gv4\" (UniqueName: \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.477988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.477911 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.514512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.514433 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gv4\" (UniqueName: \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\") pod \"payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.540879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.540611 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:19.565863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.565351 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.565863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.565442 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfj8\" (UniqueName: \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.667672 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.667628 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.668892 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.668866 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfj8\" (UniqueName: \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.669835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.668773 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.707617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.707580 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfj8\" (UniqueName: \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\") pod \"payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.811235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.811147 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:19.871032 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:19.870992 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae0b0c3_283a_413f_aa0b_27b19baf0f18.slice/crio-d9a83909aef1c5773821b007a0e59bfca4195b269361f69a2cda4a6b33eb2130 WatchSource:0}: Error finding container d9a83909aef1c5773821b007a0e59bfca4195b269361f69a2cda4a6b33eb2130: Status 404 returned error can't find the container with id d9a83909aef1c5773821b007a0e59bfca4195b269361f69a2cda4a6b33eb2130 Oct 30 20:00:19.872616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:19.872586 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:00:20.131204 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:20.131170 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3849bbd1_ded0_45b7_88e9_1960561c85fa.slice/crio-ec670ed078aa354061cfb92e76ed77d7629b41c863c8bc88f1abdd75c2466c66 WatchSource:0}: Error finding container ec670ed078aa354061cfb92e76ed77d7629b41c863c8bc88f1abdd75c2466c66: Status 404 returned error can't find the container with id ec670ed078aa354061cfb92e76ed77d7629b41c863c8bc88f1abdd75c2466c66 Oct 30 20:00:20.132322 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.132299 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:00:20.149961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.149908 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" event={"ID":"3849bbd1-ded0-45b7-88e9-1960561c85fa","Type":"ContainerStarted","Data":"ec670ed078aa354061cfb92e76ed77d7629b41c863c8bc88f1abdd75c2466c66"} Oct 30 20:00:20.156613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.156562 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" event={"ID":"7ae0b0c3-283a-413f-aa0b-27b19baf0f18","Type":"ContainerStarted","Data":"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced"} Oct 30 20:00:20.156821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.156654 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" event={"ID":"7ae0b0c3-283a-413f-aa0b-27b19baf0f18","Type":"ContainerStarted","Data":"d9a83909aef1c5773821b007a0e59bfca4195b269361f69a2cda4a6b33eb2130"} Oct 30 20:00:20.157947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.157878 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:20.203736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.203673 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" podStartSLOduration=1.203656608 podStartE2EDuration="1.203656608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.200794711 +0000 UTC m=+1487.650287437" watchObservedRunningTime="2026-10-30 20:00:20.203656608 +0000 UTC m=+1487.653149332" Oct 30 20:00:20.670710 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:20.670674 5991 scope.go:122] "RemoveContainer" containerID="f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6" Oct 30 20:00:20.894409 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:20.894353 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:00:21.168070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:21.168030 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" event={"ID":"3849bbd1-ded0-45b7-88e9-1960561c85fa","Type":"ContainerStarted","Data":"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c"} Oct 30 20:00:21.168296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:21.168104 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:21.216759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:21.216705 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" podStartSLOduration=2.216689351 podStartE2EDuration="2.216689351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:21.212586426 +0000 UTC m=+1488.662079151" watchObservedRunningTime="2026-10-30 20:00:21.216689351 +0000 UTC m=+1488.666182077" Oct 30 20:00:21.671202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:21.671160 5991 scope.go:122] "RemoveContainer" containerID="04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b" Oct 30 20:00:21.671708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:21.671429 5991 scope.go:122] "RemoveContainer" containerID="5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7" Oct 30 20:00:22.181570 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:22.181542 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerID="00872c2aee2e0a01ddf05016775fd441df562701104a6560e76ed19e0fdb274f" exitCode=60 Oct 30 20:00:22.181738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:22.181620 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerDied","Data":"00872c2aee2e0a01ddf05016775fd441df562701104a6560e76ed19e0fdb274f"} Oct 30 20:00:22.181738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:22.181691 5991 scope.go:122] "RemoveContainer" containerID="f6eb8968029615711c3c1285a77b00935f8f2fd81587fb8bede991b139ecf0e6" Oct 30 20:00:22.182219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:22.182204 5991 scope.go:122] "RemoveContainer" containerID="00872c2aee2e0a01ddf05016775fd441df562701104a6560e76ed19e0fdb274f" Oct 30 20:00:22.182653 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:22.182631 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g_odh-ai-gateway-infra(3c0a3fbf-ec23-43d7-a7cc-f052bf604305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" Oct 30 20:00:23.198236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.198138 5991 generic.go:356] "Generic (PLEG): container finished" podID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerID="33fc00afc5aeb019339b7df3c5044b09748aa46cfbf9700d61712e1d683181f8" exitCode=60 Oct 30 20:00:23.198236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.198221 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerDied","Data":"33fc00afc5aeb019339b7df3c5044b09748aa46cfbf9700d61712e1d683181f8"} Oct 30 20:00:23.198822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.198299 5991 scope.go:122] "RemoveContainer" containerID="5629f655d5d6a3ad47cb2681bfcd0f9c44f6422773d69fb1e3cf6ea6a184d4c7" Oct 30 20:00:23.198896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.198871 5991 scope.go:122] "RemoveContainer" containerID="33fc00afc5aeb019339b7df3c5044b09748aa46cfbf9700d61712e1d683181f8" Oct 30 20:00:23.199352 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:23.199316 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752_odh-ai-gateway-infra(2850e2dc-29b5-4e39-9973-19df757cdc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" Oct 30 20:00:23.200999 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.200978 5991 generic.go:356] "Generic (PLEG): container finished" podID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerID="a92f4e8dd5eaf88a370945e9bf85e34d772ec0e743aafae72c21db341e8bae73" exitCode=60 Oct 30 20:00:23.201119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.201051 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerDied","Data":"a92f4e8dd5eaf88a370945e9bf85e34d772ec0e743aafae72c21db341e8bae73"} Oct 30 20:00:23.201600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.201584 5991 scope.go:122] "RemoveContainer" containerID="a92f4e8dd5eaf88a370945e9bf85e34d772ec0e743aafae72c21db341e8bae73" Oct 30 20:00:23.202043 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:23.202020 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p_odh-ai-gateway-infra(1ff2c283-515f-410b-a98b-f59319bf2dd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" Oct 30 20:00:23.213281 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.213262 5991 scope.go:122] "RemoveContainer" containerID="04474c0729ddec41ee3d937f2c59da1251754d8cb21e67ad8c0c552dd1cf0a0b" Oct 30 20:00:23.240235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.240195 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:23.425322 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.425293 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:23.538976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.538944 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\") pod \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " Oct 30 20:00:23.539194 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.539045 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\" (UniqueName: \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\") pod \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\" (UID: \"3c0a3fbf-ec23-43d7-a7cc-f052bf604305\") " Oct 30 20:00:23.539420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.539390 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca" pod "3c0a3fbf-ec23-43d7-a7cc-f052bf604305" (UID: "3c0a3fbf-ec23-43d7-a7cc-f052bf604305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:23.539969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.539944 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.541376 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.541338 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m" pod "3c0a3fbf-ec23-43d7-a7cc-f052bf604305" (UID: "3c0a3fbf-ec23-43d7-a7cc-f052bf604305"). InnerVolumeSpecName "kube-api-access-mhv2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:23.641186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:23.641141 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhv2m\" (UniqueName: \"kubernetes.io/projected/3c0a3fbf-ec23-43d7-a7cc-f052bf604305-kube-api-access-mhv2m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.215731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.215680 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" event={"ID":"3c0a3fbf-ec23-43d7-a7cc-f052bf604305","Type":"ContainerDied","Data":"a4a5a452188b85cc953d673e2fb5990f06ca0011de484f3ee3c60b8b222fbcc1"} Oct 30 20:00:24.216173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.215762 5991 scope.go:122] "RemoveContainer" containerID="00872c2aee2e0a01ddf05016775fd441df562701104a6560e76ed19e0fdb274f" Oct 30 20:00:24.216173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.215893 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g" Oct 30 20:00:24.252735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.252701 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:24.291436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.291400 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:24.297308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.297278 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-540a43-29889840-bst8g"] Oct 30 20:00:24.304218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.304185 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:24.505724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.505691 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:24.560492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.560464 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:24.561819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.561790 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\") pod \"1ff2c283-515f-410b-a98b-f59319bf2dd1\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " Oct 30 20:00:24.561999 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.561876 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\") pod \"2850e2dc-29b5-4e39-9973-19df757cdc06\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " Oct 30 20:00:24.561999 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.561941 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\" (UniqueName: \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\") pod \"1ff2c283-515f-410b-a98b-f59319bf2dd1\" (UID: \"1ff2c283-515f-410b-a98b-f59319bf2dd1\") " Oct 30 20:00:24.562320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.562183 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca" pod "1ff2c283-515f-410b-a98b-f59319bf2dd1" (UID: "1ff2c283-515f-410b-a98b-f59319bf2dd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:24.562320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.562204 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\" (UniqueName: \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\") pod \"2850e2dc-29b5-4e39-9973-19df757cdc06\" (UID: \"2850e2dc-29b5-4e39-9973-19df757cdc06\") " Oct 30 20:00:24.563099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.563073 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2c283-515f-410b-a98b-f59319bf2dd1-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.564987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.564958 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r" pod "1ff2c283-515f-410b-a98b-f59319bf2dd1" (UID: "1ff2c283-515f-410b-a98b-f59319bf2dd1"). InnerVolumeSpecName "kube-api-access-5kv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:24.565094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.565000 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx" pod "2850e2dc-29b5-4e39-9973-19df757cdc06" (UID: "2850e2dc-29b5-4e39-9973-19df757cdc06"). InnerVolumeSpecName "kube-api-access-9rhvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:24.572094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.571542 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca" pod "2850e2dc-29b5-4e39-9973-19df757cdc06" (UID: "2850e2dc-29b5-4e39-9973-19df757cdc06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:24.665516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.665475 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2850e2dc-29b5-4e39-9973-19df757cdc06-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.665516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.665516 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kv5r\" (UniqueName: \"kubernetes.io/projected/1ff2c283-515f-410b-a98b-f59319bf2dd1-kube-api-access-5kv5r\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.665722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.665535 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhvx\" (UniqueName: \"kubernetes.io/projected/2850e2dc-29b5-4e39-9973-19df757cdc06-kube-api-access-9rhvx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.681676 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:24.681643 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" path="/var/lib/kubelet/pods/3c0a3fbf-ec23-43d7-a7cc-f052bf604305/volumes" Oct 30 20:00:25.265722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.265690 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" Oct 30 20:00:25.266205 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.265722 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752" event={"ID":"2850e2dc-29b5-4e39-9973-19df757cdc06","Type":"ContainerDied","Data":"a39c49acd70593bebd402c1ff82538ea48c7b3fa68647111280be7480baa1b43"} Oct 30 20:00:25.266205 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.265793 5991 scope.go:122] "RemoveContainer" containerID="33fc00afc5aeb019339b7df3c5044b09748aa46cfbf9700d61712e1d683181f8" Oct 30 20:00:25.267463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.267409 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" event={"ID":"1ff2c283-515f-410b-a98b-f59319bf2dd1","Type":"ContainerDied","Data":"884a07a80c9cb6321946269fa8b1f86fb44b495981ee7736cfcabf4063885a9c"} Oct 30 20:00:25.267596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.267441 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p" Oct 30 20:00:25.277398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.277335 5991 scope.go:122] "RemoveContainer" containerID="a92f4e8dd5eaf88a370945e9bf85e34d772ec0e743aafae72c21db341e8bae73" Oct 30 20:00:25.320058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.320023 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:25.324827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.324799 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889840-hv752"] Oct 30 20:00:25.366455 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.366417 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:25.371317 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:25.371291 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2224e496-29889840-msq4p"] Oct 30 20:00:26.682112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:26.682078 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" path="/var/lib/kubelet/pods/1ff2c283-515f-410b-a98b-f59319bf2dd1/volumes" Oct 30 20:00:26.682673 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:26.682657 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" path="/var/lib/kubelet/pods/2850e2dc-29b5-4e39-9973-19df757cdc06/volumes" Oct 30 20:00:32.187511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:32.187478 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:00:32.188037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:32.187553 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:00:32.957853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:32.957827 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:00:32.965762 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:32.965736 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:00:36.788541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.788510 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:00:36.789602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789578 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.789602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789601 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789636 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789643 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789660 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789667 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789681 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789689 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789724 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789731 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789754 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.789791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789762 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.789997 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790012 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790022 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790033 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790046 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790061 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790071 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.790286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.790080 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.796062 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.796042 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:36.815377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.815335 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:00:36.821679 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.821654 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ef132f\"" Oct 30 20:00:36.842511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.842469 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:00:36.843897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843875 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.843897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843897 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2850e2dc-29b5-4e39-9973-19df757cdc06" containerName="cleanup" Oct 30 20:00:36.844030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843912 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.844030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843918 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff2c283-515f-410b-a98b-f59319bf2dd1" containerName="cleanup" Oct 30 20:00:36.844030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843951 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.844030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.843956 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.844151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.844106 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0a3fbf-ec23-43d7-a7cc-f052bf604305" containerName="cleanup" Oct 30 20:00:36.848076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.848057 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:36.866014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.865979 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:00:36.873906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.872774 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ef132f-dockercfg-ttrdq\"" Oct 30 20:00:36.873906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.874575 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:36.873906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.874771 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h87c\" (UniqueName: \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:36.985387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.982743 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tq5\" (UniqueName: \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:36.985387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.982821 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:36.985387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.982994 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:36.985387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.983185 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h87c\" (UniqueName: \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:36.985387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:36.984477 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:37.053466 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.048197 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h87c\" (UniqueName: \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\") pod \"payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:37.085311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.085270 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tq5\" (UniqueName: \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:37.085959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.085918 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:37.090308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.090275 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:37.111386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.108157 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tq5\" (UniqueName: \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\") pod \"payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:37.112474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.111999 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:37.166227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.166179 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:37.476878 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:37.476842 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50d6d58_4921_4961_b36c_a2158faa1b26.slice/crio-573a3995683d82b1906fe90266ed653a442397cfb5dabf1e775694e4a8554fda WatchSource:0}: Error finding container 573a3995683d82b1906fe90266ed653a442397cfb5dabf1e775694e4a8554fda: Status 404 returned error can't find the container with id 573a3995683d82b1906fe90266ed653a442397cfb5dabf1e775694e4a8554fda Oct 30 20:00:37.479352 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.479318 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:00:37.571655 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:00:37.571607 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e934b_9f7b_4ded_b54b_88c43a11a3fa.slice/crio-0242c7d9dc24f245b6829a7ffdc7c311052881f315eecf26727f30895a4358c3 WatchSource:0}: Error finding container 0242c7d9dc24f245b6829a7ffdc7c311052881f315eecf26727f30895a4358c3: Status 404 returned error can't find the container with id 0242c7d9dc24f245b6829a7ffdc7c311052881f315eecf26727f30895a4358c3 Oct 30 20:00:37.574648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:37.574621 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:00:38.429531 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.429493 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" event={"ID":"a50d6d58-4921-4961-b36c-a2158faa1b26","Type":"ContainerStarted","Data":"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527"} Oct 30 20:00:38.430001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.429557 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" event={"ID":"a50d6d58-4921-4961-b36c-a2158faa1b26","Type":"ContainerStarted","Data":"573a3995683d82b1906fe90266ed653a442397cfb5dabf1e775694e4a8554fda"} Oct 30 20:00:38.430001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.429585 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:38.431351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.431317 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" event={"ID":"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa","Type":"ContainerStarted","Data":"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e"} Oct 30 20:00:38.431484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.431401 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:38.431484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.431419 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" event={"ID":"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa","Type":"ContainerStarted","Data":"0242c7d9dc24f245b6829a7ffdc7c311052881f315eecf26727f30895a4358c3"} Oct 30 20:00:38.473842 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.473789 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" podStartSLOduration=2.473775277 podStartE2EDuration="2.473775277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:38.47031262 +0000 UTC m=+1505.919805346" watchObservedRunningTime="2026-10-30 20:00:38.473775277 +0000 UTC m=+1505.923268002" Oct 30 20:00:38.515058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:38.515007 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" podStartSLOduration=2.514992359 podStartE2EDuration="2.514992359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:38.510170166 +0000 UTC m=+1505.959662882" watchObservedRunningTime="2026-10-30 20:00:38.514992359 +0000 UTC m=+1505.964485124" Oct 30 20:00:41.184197 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:41.184159 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:00:46.989399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:46.987017 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 20:00:46.989399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:46.987269 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" podUID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" containerName="maas-api" containerID="cri-o://f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17" gracePeriod=30 Oct 30 20:00:47.415385 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.415327 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 20:00:47.515189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.515116 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\") pod \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " Oct 30 20:00:47.515481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.515445 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\" (UniqueName: \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\") pod \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " Oct 30 20:00:47.515621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.515603 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\") pod \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " Oct 30 20:00:47.515709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.515695 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\") pod \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\" (UID: \"3c3280a4-8496-4a2e-bc07-8d11b0de51fe\") " Oct 30 20:00:47.516072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.516044 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca" pod "3c3280a4-8496-4a2e-bc07-8d11b0de51fe" (UID: "3c3280a4-8496-4a2e-bc07-8d11b0de51fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:47.518132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.516491 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.519431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.519351 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls" pod "3c3280a4-8496-4a2e-bc07-8d11b0de51fe" (UID: "3c3280a4-8496-4a2e-bc07-8d11b0de51fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:47.519703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.519679 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p" pod "3c3280a4-8496-4a2e-bc07-8d11b0de51fe" (UID: "3c3280a4-8496-4a2e-bc07-8d11b0de51fe"). InnerVolumeSpecName "kube-api-access-6w66p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:47.530382 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.530327 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" containerID="f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17" exitCode=0 Oct 30 20:00:47.530849 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.530822 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" event={"ID":"3c3280a4-8496-4a2e-bc07-8d11b0de51fe","Type":"ContainerDied","Data":"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17"} Oct 30 20:00:47.531121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.531058 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" event={"ID":"3c3280a4-8496-4a2e-bc07-8d11b0de51fe","Type":"ContainerDied","Data":"dd10eabffb9a2c560f7d1279be50c28f00a089baac533884e5ec72f36540e92b"} Oct 30 20:00:47.544685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.536785 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs" pod "3c3280a4-8496-4a2e-bc07-8d11b0de51fe" (UID: "3c3280a4-8496-4a2e-bc07-8d11b0de51fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:47.547731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.536817 5991 scope.go:122] "RemoveContainer" containerID="f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17" Oct 30 20:00:47.548005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.536903 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g" Oct 30 20:00:47.616550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.616524 5991 scope.go:122] "RemoveContainer" containerID="f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17" Oct 30 20:00:47.616982 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:47.616951 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17\": container with ID starting with f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17 not found: ID does not exist" containerID="f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17" Oct 30 20:00:47.617102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.617002 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17"} err="failed to get container status \"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17\": rpc error: code = NotFound desc = could not find container \"f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17\": container with ID starting with f16f009477d28712cda056c0a8014948f65a1e001b1ab2b0ffdfe2eab0b5ed17 not found: ID does not exist" Oct 30 20:00:47.618603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.618520 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w66p\" (UniqueName: \"kubernetes.io/projected/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-kube-api-access-6w66p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.618603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.618543 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.618603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.618552 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3280a4-8496-4a2e-bc07-8d11b0de51fe-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.637547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.637500 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 20:00:47.642287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:47.642250 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93263c-6b9696fff5-r249g"] Oct 30 20:00:48.681529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:48.681493 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" path="/var/lib/kubelet/pods/3c3280a4-8496-4a2e-bc07-8d11b0de51fe/volumes" Oct 30 20:00:49.468392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:49.459293 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:00:49.468392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:49.459496 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:00:50.870608 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:50.870571 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:00:56.969602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.969560 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:00:56.970989 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.970964 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" containerName="maas-api" Oct 30 20:00:56.970989 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.970990 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" containerName="maas-api" Oct 30 20:00:56.971251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.971239 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3280a4-8496-4a2e-bc07-8d11b0de51fe" containerName="maas-api" Oct 30 20:00:56.975152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.975132 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:56.992970 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.992935 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:00:56.999624 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:56.999593 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-652137-serving-cert\"" Oct 30 20:00:57.013512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.013448 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvr2\" (UniqueName: \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.013827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.013619 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.013827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.013791 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.013949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.013878 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.115598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.115559 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.115810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.115658 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.115881 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:57.115796 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-652137-serving-cert: secret "maas-api-e2e-api-b-652137-serving-cert" not found Oct 30 20:00:57.115881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.115831 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvr2\" (UniqueName: \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.115881 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:00:57.115861 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls podName:f76498a3-61c2-4e27-aa37-a19caa4a2f83 nodeName:}" failed. No retries permitted until 2026-10-30 20:00:57.615842818 +0000 UTC m=+1525.065335521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls") pod "maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" (UID: "f76498a3-61c2-4e27-aa37-a19caa4a2f83") : secret "maas-api-e2e-api-b-652137-serving-cert" not found Oct 30 20:00:57.116046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.115978 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.116756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.116731 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.118120 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.118094 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.138704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.138671 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvr2\" (UniqueName: \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.625035 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.624996 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.627541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.627512 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") pod \"maas-api-e2e-api-b-652137-56bc556b9b-nnzfz\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:57.892496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:57.892395 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:58.217141 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:58.216609 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:00:58.654091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:58.654062 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" event={"ID":"f76498a3-61c2-4e27-aa37-a19caa4a2f83","Type":"ContainerStarted","Data":"bc171557c4c5d39a1db40eedaa4f676524209567b19eccde65126871bb9e5917"} Oct 30 20:00:59.665591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:59.665554 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" event={"ID":"f76498a3-61c2-4e27-aa37-a19caa4a2f83","Type":"ContainerStarted","Data":"1857832849bb703c515e859dc0bcdef088a8bc2a83e5ef12b19a8f8de0f2c85f"} Oct 30 20:00:59.666071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:59.665663 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:00:59.709147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:00:59.709083 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" podStartSLOduration=3.305854534 podStartE2EDuration="3.709062428s" totalImagesPullingTime="403.207894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:59.706506825 +0000 UTC m=+1527.155999563" watchObservedRunningTime="2026-10-30 20:00:59.709062428 +0000 UTC m=+1527.158555199" Oct 30 20:01:02.160414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:02.160378 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 20:01:05.694505 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:05.694469 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:01:06.717745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:06.717713 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 20:01:06.751696 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:06.751660 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 20:01:07.160959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.160915 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" podUID="03908caf-8fc5-4f4d-9996-e9ae0868b912" containerName="payload-pre-processing" containerID="cri-o://46d948fac3df18530905239ec04732ed08781671515b0c2e76ccc7452e5c7b16" gracePeriod=25 Oct 30 20:01:07.767134 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.767083 5991 generic.go:356] "Generic (PLEG): container finished" podID="03908caf-8fc5-4f4d-9996-e9ae0868b912" containerID="46d948fac3df18530905239ec04732ed08781671515b0c2e76ccc7452e5c7b16" exitCode=0 Oct 30 20:01:07.767859 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.767833 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" event={"ID":"03908caf-8fc5-4f4d-9996-e9ae0868b912","Type":"ContainerDied","Data":"46d948fac3df18530905239ec04732ed08781671515b0c2e76ccc7452e5c7b16"} Oct 30 20:01:07.768095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.768057 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" event={"ID":"03908caf-8fc5-4f4d-9996-e9ae0868b912","Type":"ContainerDied","Data":"68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4"} Oct 30 20:01:07.768234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.768220 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68cce9bdf3374af615488d2ade85f8a35f54ed8fa4e918f1b0de4ffe95ea85f4" Oct 30 20:01:07.769665 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.769642 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 20:01:07.934588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.934495 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\" (UniqueName: \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\") pod \"03908caf-8fc5-4f4d-9996-e9ae0868b912\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " Oct 30 20:01:07.935509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.935481 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\") pod \"03908caf-8fc5-4f4d-9996-e9ae0868b912\" (UID: \"03908caf-8fc5-4f4d-9996-e9ae0868b912\") " Oct 30 20:01:07.936208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.936177 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume" pod "03908caf-8fc5-4f4d-9996-e9ae0868b912" (UID: "03908caf-8fc5-4f4d-9996-e9ae0868b912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:07.936715 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.936690 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03908caf-8fc5-4f4d-9996-e9ae0868b912-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:07.938497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:07.938463 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br" pod "03908caf-8fc5-4f4d-9996-e9ae0868b912" (UID: "03908caf-8fc5-4f4d-9996-e9ae0868b912"). InnerVolumeSpecName "kube-api-access-5x5br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:08.039910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:08.039872 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5br\" (UniqueName: \"kubernetes.io/projected/03908caf-8fc5-4f4d-9996-e9ae0868b912-kube-api-access-5x5br\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.803350 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:08.803307 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z" Oct 30 20:01:08.859711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:08.859676 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 20:01:08.891302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:08.891250 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d96913-7dbbd674bc-dxt4z"] Oct 30 20:01:10.682583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:10.682548 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03908caf-8fc5-4f4d-9996-e9ae0868b912" path="/var/lib/kubelet/pods/03908caf-8fc5-4f4d-9996-e9ae0868b912/volumes" Oct 30 20:01:11.718956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:11.718908 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" podUID="fe8bad11-f317-4812-bb49-26a34220c5b4" containerName="payload-processing" containerID="cri-o://447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829" gracePeriod=25 Oct 30 20:01:11.752614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:11.752551 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" podUID="232954cc-e118-44ec-8e48-5f01f89837ee" containerName="payload-pre-processing" containerID="cri-o://aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9" gracePeriod=25 Oct 30 20:01:11.866168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:11.865596 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:01:11.866168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:11.865716 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:12.094721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.094688 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 20:01:12.146236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.146204 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 20:01:12.213380 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.213324 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\" (UniqueName: \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\") pod \"232954cc-e118-44ec-8e48-5f01f89837ee\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " Oct 30 20:01:12.213631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.213611 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\" (UniqueName: \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\") pod \"fe8bad11-f317-4812-bb49-26a34220c5b4\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " Oct 30 20:01:12.213739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.213718 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\") pod \"fe8bad11-f317-4812-bb49-26a34220c5b4\" (UID: \"fe8bad11-f317-4812-bb49-26a34220c5b4\") " Oct 30 20:01:12.213981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.213961 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\") pod \"232954cc-e118-44ec-8e48-5f01f89837ee\" (UID: \"232954cc-e118-44ec-8e48-5f01f89837ee\") " Oct 30 20:01:12.214345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.214320 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume" pod "232954cc-e118-44ec-8e48-5f01f89837ee" (UID: "232954cc-e118-44ec-8e48-5f01f89837ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:12.214471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.214320 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume" pod "fe8bad11-f317-4812-bb49-26a34220c5b4" (UID: "fe8bad11-f317-4812-bb49-26a34220c5b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:12.214821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.214803 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8bad11-f317-4812-bb49-26a34220c5b4-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:12.214899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.214824 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232954cc-e118-44ec-8e48-5f01f89837ee-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:12.216169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.216145 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh" pod "232954cc-e118-44ec-8e48-5f01f89837ee" (UID: "232954cc-e118-44ec-8e48-5f01f89837ee"). InnerVolumeSpecName "kube-api-access-6vfwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:12.216344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.216325 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq" pod "fe8bad11-f317-4812-bb49-26a34220c5b4" (UID: "fe8bad11-f317-4812-bb49-26a34220c5b4"). InnerVolumeSpecName "kube-api-access-nfrhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:12.317121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.317033 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfwh\" (UniqueName: \"kubernetes.io/projected/232954cc-e118-44ec-8e48-5f01f89837ee-kube-api-access-6vfwh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:12.317121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.317070 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrhq\" (UniqueName: \"kubernetes.io/projected/fe8bad11-f317-4812-bb49-26a34220c5b4-kube-api-access-nfrhq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:12.843279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.843242 5991 generic.go:356] "Generic (PLEG): container finished" podID="232954cc-e118-44ec-8e48-5f01f89837ee" containerID="aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9" exitCode=0 Oct 30 20:01:12.843790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.843286 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" Oct 30 20:01:12.843790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.843336 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" event={"ID":"232954cc-e118-44ec-8e48-5f01f89837ee","Type":"ContainerDied","Data":"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9"} Oct 30 20:01:12.843790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.843411 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh" event={"ID":"232954cc-e118-44ec-8e48-5f01f89837ee","Type":"ContainerDied","Data":"6cb471a9cf05b964a8ae53e794771580a14719a1c15d992ee452339b7ca1d925"} Oct 30 20:01:12.843790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.843440 5991 scope.go:122] "RemoveContainer" containerID="aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9" Oct 30 20:01:12.845849 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.845808 5991 generic.go:356] "Generic (PLEG): container finished" podID="fe8bad11-f317-4812-bb49-26a34220c5b4" containerID="447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829" exitCode=0 Oct 30 20:01:12.845979 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.845901 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" event={"ID":"fe8bad11-f317-4812-bb49-26a34220c5b4","Type":"ContainerDied","Data":"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829"} Oct 30 20:01:12.845979 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.845938 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" event={"ID":"fe8bad11-f317-4812-bb49-26a34220c5b4","Type":"ContainerDied","Data":"6a20f19857af55a70342d64cf578c7fc3504f070c054f5e250438d454140f375"} Oct 30 20:01:12.846099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.846007 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk" Oct 30 20:01:12.858772 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.858748 5991 scope.go:122] "RemoveContainer" containerID="aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9" Oct 30 20:01:12.859113 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:12.859090 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9\": container with ID starting with aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9 not found: ID does not exist" containerID="aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9" Oct 30 20:01:12.859171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.859121 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9"} err="failed to get container status \"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9\": rpc error: code = NotFound desc = could not find container \"aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9\": container with ID starting with aabd91332582273885124057c6ef3e7307db8cd267cc77cdccbd9997530854e9 not found: ID does not exist" Oct 30 20:01:12.859171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.859140 5991 scope.go:122] "RemoveContainer" containerID="447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829" Oct 30 20:01:12.870137 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.870112 5991 scope.go:122] "RemoveContainer" containerID="447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829" Oct 30 20:01:12.870589 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:12.870566 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829\": container with ID starting with 447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829 not found: ID does not exist" containerID="447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829" Oct 30 20:01:12.870668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.870603 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829"} err="failed to get container status \"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829\": rpc error: code = NotFound desc = could not find container \"447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829\": container with ID starting with 447b84698de6a04b87a2664bafdaf87a6e432818dba8f6e251a2bf250352d829 not found: ID does not exist" Oct 30 20:01:12.902744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.902702 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 20:01:12.914557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.914525 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-540a43-5596b55cf4-w8wzk"] Oct 30 20:01:12.951162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.951079 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 20:01:12.965912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:12.965872 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-540a43-65cc6b4595-2jrdh"] Oct 30 20:01:13.873881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.873775 5991 generic.go:356] "Generic (PLEG): container finished" podID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerID="9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a" exitCode=0 Oct 30 20:01:13.873881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.873831 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerDied","Data":"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a"} Oct 30 20:01:13.874541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.874484 5991 scope.go:122] "RemoveContainer" containerID="9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a" Oct 30 20:01:13.877196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.877157 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:13.877347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.877258 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:13.877347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:13.877305 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:14.787479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:14.782022 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232954cc-e118-44ec-8e48-5f01f89837ee" path="/var/lib/kubelet/pods/232954cc-e118-44ec-8e48-5f01f89837ee/volumes" Oct 30 20:01:14.787479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:14.782886 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8bad11-f317-4812-bb49-26a34220c5b4" path="/var/lib/kubelet/pods/fe8bad11-f317-4812-bb49-26a34220c5b4/volumes" Oct 30 20:01:14.937390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:14.925306 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerStarted","Data":"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502"} Oct 30 20:01:15.517919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:15.517882 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:01:15.518300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:15.518274 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:15.518508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:15.518479 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:16.252788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.252484 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:01:16.473391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.473318 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.478397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.476245 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.511460 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.509211 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:16.511460 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.509286 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:16.551387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.550705 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.555402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.553477 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.584391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.579603 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.584391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.581512 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.663390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.660507 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 20:01:16.687387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.685785 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 20:01:16.912494 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:16.910213 5991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh" podUID="0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 20:01:17.507350 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:17.507299 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:17.507836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:17.507417 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:18.507493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:18.507455 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:18.507941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:18.507512 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:19.507169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:19.507133 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:19.507430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:19.507194 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:20.507313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:20.507272 5991 patch_prober.go:29] interesting pod/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:01:20.507786 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:20.507343 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:01:20.871941 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:20.871848 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:01:21.512084 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:21.512052 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:01:21.512621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:21.512578 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:01:21.513223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:21.513201 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:01:44.682876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.682838 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:01:44.683736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683718 5991 cpu_manager.go:414] "containerMap: removing container" podUID="232954cc-e118-44ec-8e48-5f01f89837ee" containerName="payload-pre-processing" Oct 30 20:01:44.683818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683738 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232954cc-e118-44ec-8e48-5f01f89837ee" containerName="payload-pre-processing" Oct 30 20:01:44.683818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683773 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fe8bad11-f317-4812-bb49-26a34220c5b4" containerName="payload-processing" Oct 30 20:01:44.683818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683779 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8bad11-f317-4812-bb49-26a34220c5b4" containerName="payload-processing" Oct 30 20:01:44.683818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683809 5991 cpu_manager.go:414] "containerMap: removing container" podUID="03908caf-8fc5-4f4d-9996-e9ae0868b912" containerName="payload-pre-processing" Oct 30 20:01:44.683818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683815 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03908caf-8fc5-4f4d-9996-e9ae0868b912" containerName="payload-pre-processing" Oct 30 20:01:44.683978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683939 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="03908caf-8fc5-4f4d-9996-e9ae0868b912" containerName="payload-pre-processing" Oct 30 20:01:44.683978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683953 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="232954cc-e118-44ec-8e48-5f01f89837ee" containerName="payload-pre-processing" Oct 30 20:01:44.683978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.683965 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8bad11-f317-4812-bb49-26a34220c5b4" containerName="payload-processing" Oct 30 20:01:44.689780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.689751 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.711978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.711941 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:01:44.723598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.723563 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2224e496-serving-cert\"" Oct 30 20:01:44.769292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.769259 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.769506 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.769440 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.769569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.769544 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.769715 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.769698 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kf9p\" (UniqueName: \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.871554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.871512 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.871756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.871680 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kf9p\" (UniqueName: \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.871826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.871809 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.871953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.871936 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.872669 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.872643 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.874168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.874137 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.874339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.874321 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:44.897102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:44.897056 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kf9p\" (UniqueName: \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\") pod \"maas-api-e2e-mt-2224e496-d58664b78-r5l4p\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:45.010445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:45.010403 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:45.345164 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:01:45.345125 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4972ae4e_0a8d_44be_9511_1f307748b48c.slice/crio-88ea35811e446500841dd3c8a45cbe7a835e84f8dca2a4a076eca7bd2e45c4df WatchSource:0}: Error finding container 88ea35811e446500841dd3c8a45cbe7a835e84f8dca2a4a076eca7bd2e45c4df: Status 404 returned error can't find the container with id 88ea35811e446500841dd3c8a45cbe7a835e84f8dca2a4a076eca7bd2e45c4df Oct 30 20:01:45.346610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:45.346586 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:01:45.610470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:45.610358 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" event={"ID":"4972ae4e-0a8d-44be-9511-1f307748b48c","Type":"ContainerStarted","Data":"88ea35811e446500841dd3c8a45cbe7a835e84f8dca2a4a076eca7bd2e45c4df"} Oct 30 20:01:46.620399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:46.620345 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" event={"ID":"4972ae4e-0a8d-44be-9511-1f307748b48c","Type":"ContainerStarted","Data":"9149bf379ddb4d9a93c5f88bb7eff28e0f2221e8b686dbf998ea04e207765597"} Oct 30 20:01:46.620843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:46.620618 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:46.668215 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:46.668157 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" podStartSLOduration=2.179611444 podStartE2EDuration="2.668140855s" totalImagesPullingTime="488.529411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:46.663623752 +0000 UTC m=+1574.113116479" watchObservedRunningTime="2026-10-30 20:01:46.668140855 +0000 UTC m=+1574.117633580" Oct 30 20:01:50.871813 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:50.871771 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:01:51.231283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:51.231243 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 20:01:51.262695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:51.262661 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 20:01:52.636696 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:52.636667 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:01:56.232025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.231975 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" podUID="c3708da6-b791-440b-a283-df0be15c2257" containerName="payload-processing" containerID="cri-o://6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061" gracePeriod=25 Oct 30 20:01:56.263342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.263283 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" podUID="a77ff384-212f-45ad-8b80-28a62b255b6f" containerName="payload-pre-processing" containerID="cri-o://3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf" gracePeriod=25 Oct 30 20:01:56.360462 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:56.360418 5991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77ff384_212f_45ad_8b80_28a62b255b6f.slice/crio-conmon-3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77ff384_212f_45ad_8b80_28a62b255b6f.slice/crio-3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:01:56.576176 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.576139 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 20:01:56.601377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.601336 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\") pod \"c3708da6-b791-440b-a283-df0be15c2257\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " Oct 30 20:01:56.601616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.601595 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\" (UniqueName: \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\") pod \"c3708da6-b791-440b-a283-df0be15c2257\" (UID: \"c3708da6-b791-440b-a283-df0be15c2257\") " Oct 30 20:01:56.601821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.601746 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume" pod "c3708da6-b791-440b-a283-df0be15c2257" (UID: "c3708da6-b791-440b-a283-df0be15c2257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:56.602532 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.602507 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3708da6-b791-440b-a283-df0be15c2257-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.603974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.603853 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7" pod "c3708da6-b791-440b-a283-df0be15c2257" (UID: "c3708da6-b791-440b-a283-df0be15c2257"). InnerVolumeSpecName "kube-api-access-zvsf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:56.628650 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.628594 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 20:01:56.703557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.703524 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\") pod \"a77ff384-212f-45ad-8b80-28a62b255b6f\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " Oct 30 20:01:56.703969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.703944 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\" (UniqueName: \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\") pod \"a77ff384-212f-45ad-8b80-28a62b255b6f\" (UID: \"a77ff384-212f-45ad-8b80-28a62b255b6f\") " Oct 30 20:01:56.704076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.703972 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume" pod "a77ff384-212f-45ad-8b80-28a62b255b6f" (UID: "a77ff384-212f-45ad-8b80-28a62b255b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:56.704920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.704810 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77ff384-212f-45ad-8b80-28a62b255b6f-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.704920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.704833 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvsf7\" (UniqueName: \"kubernetes.io/projected/c3708da6-b791-440b-a283-df0be15c2257-kube-api-access-zvsf7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.707421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.707392 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd" pod "a77ff384-212f-45ad-8b80-28a62b255b6f" (UID: "a77ff384-212f-45ad-8b80-28a62b255b6f"). InnerVolumeSpecName "kube-api-access-dfdtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:56.724223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.724190 5991 generic.go:356] "Generic (PLEG): container finished" podID="c3708da6-b791-440b-a283-df0be15c2257" containerID="6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061" exitCode=0 Oct 30 20:01:56.724384 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.724235 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" Oct 30 20:01:56.724384 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.724270 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" event={"ID":"c3708da6-b791-440b-a283-df0be15c2257","Type":"ContainerDied","Data":"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061"} Oct 30 20:01:56.724384 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.724323 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m" event={"ID":"c3708da6-b791-440b-a283-df0be15c2257","Type":"ContainerDied","Data":"95aa59bfc2b7419c7a4b15dd6e9ea312b737337af830232d8b9f7812c137703e"} Oct 30 20:01:56.724384 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.724346 5991 scope.go:122] "RemoveContainer" containerID="6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061" Oct 30 20:01:56.726292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.726265 5991 generic.go:356] "Generic (PLEG): container finished" podID="a77ff384-212f-45ad-8b80-28a62b255b6f" containerID="3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf" exitCode=0 Oct 30 20:01:56.726488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.726301 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" event={"ID":"a77ff384-212f-45ad-8b80-28a62b255b6f","Type":"ContainerDied","Data":"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf"} Oct 30 20:01:56.726488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.726343 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" event={"ID":"a77ff384-212f-45ad-8b80-28a62b255b6f","Type":"ContainerDied","Data":"d4adfa1fbb13d10472977cc63f8710bfe70ad6b45afad70f0e699dd7227813b1"} Oct 30 20:01:56.726488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.726346 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg" Oct 30 20:01:56.736110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.736082 5991 scope.go:122] "RemoveContainer" containerID="6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061" Oct 30 20:01:56.736425 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:56.736408 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061\": container with ID starting with 6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061 not found: ID does not exist" containerID="6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061" Oct 30 20:01:56.736478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.736434 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061"} err="failed to get container status \"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061\": rpc error: code = NotFound desc = could not find container \"6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061\": container with ID starting with 6e1e62adc636bf607038d32c75293ade2afade8b1dadb44818b853cff01f9061 not found: ID does not exist" Oct 30 20:01:56.736478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.736451 5991 scope.go:122] "RemoveContainer" containerID="3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf" Oct 30 20:01:56.747063 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.747042 5991 scope.go:122] "RemoveContainer" containerID="3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf" Oct 30 20:01:56.747398 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:01:56.747374 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf\": container with ID starting with 3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf not found: ID does not exist" containerID="3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf" Oct 30 20:01:56.747473 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.747408 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf"} err="failed to get container status \"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf\": rpc error: code = NotFound desc = could not find container \"3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf\": container with ID starting with 3d9bb3891e2da95cbf9025cd945bfd524f4b3623483a2d6c7ab96839baa402cf not found: ID does not exist" Oct 30 20:01:56.777476 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.777375 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 20:01:56.781563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.781537 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa42c3-6bc79cd766-8mt2m"] Oct 30 20:01:56.807395 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.807343 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdtd\" (UniqueName: \"kubernetes.io/projected/a77ff384-212f-45ad-8b80-28a62b255b6f-kube-api-access-dfdtd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.826146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.824462 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 20:01:56.828553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:56.828520 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aa42c3-7499c48dff-6c6wg"] Oct 30 20:01:58.681798 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:58.681764 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77ff384-212f-45ad-8b80-28a62b255b6f" path="/var/lib/kubelet/pods/a77ff384-212f-45ad-8b80-28a62b255b6f/volumes" Oct 30 20:01:58.682248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:01:58.682233 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3708da6-b791-440b-a283-df0be15c2257" path="/var/lib/kubelet/pods/c3708da6-b791-440b-a283-df0be15c2257/volumes" Oct 30 20:02:01.931436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:01.931405 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:02:12.467472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.467434 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 20:02:12.467917 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.467679 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" podUID="8d573997-3994-4a35-af27-bbc80c1b7c26" containerName="maas-api" containerID="cri-o://5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d" gracePeriod=30 Oct 30 20:02:12.836802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.836774 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 20:02:12.968095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.968056 5991 generic.go:356] "Generic (PLEG): container finished" podID="8d573997-3994-4a35-af27-bbc80c1b7c26" containerID="5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d" exitCode=0 Oct 30 20:02:12.968310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.968125 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" Oct 30 20:02:12.968310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.968260 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" event={"ID":"8d573997-3994-4a35-af27-bbc80c1b7c26","Type":"ContainerDied","Data":"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d"} Oct 30 20:02:12.968310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.968299 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz" event={"ID":"8d573997-3994-4a35-af27-bbc80c1b7c26","Type":"ContainerDied","Data":"d427843db961eb25befd30e035920f624751076f743fb2e547abf3ad4662cee5"} Oct 30 20:02:12.968530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.968327 5991 scope.go:122] "RemoveContainer" containerID="5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d" Oct 30 20:02:12.984084 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.984058 5991 scope.go:122] "RemoveContainer" containerID="5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d" Oct 30 20:02:12.984494 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:12.984455 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d\": container with ID starting with 5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d not found: ID does not exist" containerID="5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d" Oct 30 20:02:12.984623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:12.984497 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d"} err="failed to get container status \"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d\": rpc error: code = NotFound desc = could not find container \"5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d\": container with ID starting with 5708a8093f3fde6e2c8178ba8cf8baccda2ca312b30841ff4b52a2573e0bcd8d not found: ID does not exist" Oct 30 20:02:13.029816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.027854 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\" (UniqueName: \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\") pod \"8d573997-3994-4a35-af27-bbc80c1b7c26\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " Oct 30 20:02:13.029816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.028064 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\") pod \"8d573997-3994-4a35-af27-bbc80c1b7c26\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " Oct 30 20:02:13.029816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.028135 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") pod \"8d573997-3994-4a35-af27-bbc80c1b7c26\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " Oct 30 20:02:13.029816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.028313 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\") pod \"8d573997-3994-4a35-af27-bbc80c1b7c26\" (UID: \"8d573997-3994-4a35-af27-bbc80c1b7c26\") " Oct 30 20:02:13.030744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.030693 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca" pod "8d573997-3994-4a35-af27-bbc80c1b7c26" (UID: "8d573997-3994-4a35-af27-bbc80c1b7c26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:13.043498 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.043456 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs" pod "8d573997-3994-4a35-af27-bbc80c1b7c26" (UID: "8d573997-3994-4a35-af27-bbc80c1b7c26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:13.054941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.054761 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls" pod "8d573997-3994-4a35-af27-bbc80c1b7c26" (UID: "8d573997-3994-4a35-af27-bbc80c1b7c26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:13.054941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.054907 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw" pod "8d573997-3994-4a35-af27-bbc80c1b7c26" (UID: "8d573997-3994-4a35-af27-bbc80c1b7c26"). InnerVolumeSpecName "kube-api-access-zhjkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:13.130913 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.130876 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d573997-3994-4a35-af27-bbc80c1b7c26-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:13.130913 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.130909 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhjkw\" (UniqueName: \"kubernetes.io/projected/8d573997-3994-4a35-af27-bbc80c1b7c26-kube-api-access-zhjkw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:13.131182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.130924 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:13.131182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.130936 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d573997-3994-4a35-af27-bbc80c1b7c26-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:13.319655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.319574 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 20:02:13.331817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.331788 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d96913-5dcbc6944-fbbrz"] Oct 30 20:02:13.725272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.725234 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:02:13.726131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726116 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c3708da6-b791-440b-a283-df0be15c2257" containerName="payload-processing" Oct 30 20:02:13.726181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726132 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3708da6-b791-440b-a283-df0be15c2257" containerName="payload-processing" Oct 30 20:02:13.726181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726147 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8d573997-3994-4a35-af27-bbc80c1b7c26" containerName="maas-api" Oct 30 20:02:13.726181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726153 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d573997-3994-4a35-af27-bbc80c1b7c26" containerName="maas-api" Oct 30 20:02:13.726181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726172 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a77ff384-212f-45ad-8b80-28a62b255b6f" containerName="payload-pre-processing" Oct 30 20:02:13.726181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726177 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77ff384-212f-45ad-8b80-28a62b255b6f" containerName="payload-pre-processing" Oct 30 20:02:13.726346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726316 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3708da6-b791-440b-a283-df0be15c2257" containerName="payload-processing" Oct 30 20:02:13.726346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726324 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d573997-3994-4a35-af27-bbc80c1b7c26" containerName="maas-api" Oct 30 20:02:13.726346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.726333 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77ff384-212f-45ad-8b80-28a62b255b6f" containerName="payload-pre-processing" Oct 30 20:02:13.730776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.730752 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.759528 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.759493 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-652137-openshift-default-dockercfg-6plw8\"" Oct 30 20:02:13.761561 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.761522 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:02:13.841943 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.841899 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.841967 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842057 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842101 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842198 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842294 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtwb\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842487 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842549 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.842622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.842607 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.944711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944673 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.944935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944742 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.944935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944792 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.944935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944844 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.944935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944884 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.944949 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945019 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945080 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945161 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtwb\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945218 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945155 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945255 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945657 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.945829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.945807 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.947666 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.947637 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.948169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.948135 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.965308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.965275 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:13.965926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:13.965898 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtwb\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:14.046597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.046510 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:14.394314 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:02:14.394273 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cb8bc2_1960_442e_a13c_5c8b2f895bc3.slice/crio-bb191f2063e7d22fdc5b4f36e5390998d1da729dca2bd8205b1b219b4aa0ca1f WatchSource:0}: Error finding container bb191f2063e7d22fdc5b4f36e5390998d1da729dca2bd8205b1b219b4aa0ca1f: Status 404 returned error can't find the container with id bb191f2063e7d22fdc5b4f36e5390998d1da729dca2bd8205b1b219b4aa0ca1f Oct 30 20:02:14.396868 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.396843 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:02:14.397600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.397561 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:14.397665 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.397640 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:14.397707 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.397675 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:14.682303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.682218 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d573997-3994-4a35-af27-bbc80c1b7c26" path="/var/lib/kubelet/pods/8d573997-3994-4a35-af27-bbc80c1b7c26/volumes" Oct 30 20:02:14.989942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.989905 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" event={"ID":"50cb8bc2-1960-442e-a13c-5c8b2f895bc3","Type":"ContainerStarted","Data":"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043"} Oct 30 20:02:14.990452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:14.989968 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" event={"ID":"50cb8bc2-1960-442e-a13c-5c8b2f895bc3","Type":"ContainerStarted","Data":"bb191f2063e7d22fdc5b4f36e5390998d1da729dca2bd8205b1b219b4aa0ca1f"} Oct 30 20:02:15.046704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.046669 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:15.049018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.048990 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:15.049172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.049069 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:15.052536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.052477 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podStartSLOduration=2.052463678 podStartE2EDuration="2.052463678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:15.050427538 +0000 UTC m=+1602.499920265" watchObservedRunningTime="2026-10-30 20:02:15.052463678 +0000 UTC m=+1602.501956402" Oct 30 20:02:15.859998 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.859962 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:02:15.897204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.897145 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:02:15.897470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.897281 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:15.913130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.913100 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7b5635-dockercfg-nlcb6\"" Oct 30 20:02:15.935823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.935055 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7b5635\"" Oct 30 20:02:16.011406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.996694 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.011406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:15.997083 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwbg\" (UniqueName: \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.072388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.068817 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:16.205653 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.205543 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:16.205948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.098597 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwbg\" (UniqueName: \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.206345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.206320 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.227399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.224740 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwbg\" (UniqueName: \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.235390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.228575 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.521592 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.521556 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:16.862082 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:02:16.862048 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918edd0c_ea25_4710_8d80_2ee6616501a9.slice/crio-2c660c16e9111c4bcf4631ae0138151f38a86eb3d474049fd17ceefb61c5f472 WatchSource:0}: Error finding container 2c660c16e9111c4bcf4631ae0138151f38a86eb3d474049fd17ceefb61c5f472: Status 404 returned error can't find the container with id 2c660c16e9111c4bcf4631ae0138151f38a86eb3d474049fd17ceefb61c5f472 Oct 30 20:02:16.862207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:16.862117 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:02:17.047469 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.047426 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:17.047922 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.047485 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:17.214551 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.214455 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" event={"ID":"918edd0c-ea25-4710-8d80-2ee6616501a9","Type":"ContainerStarted","Data":"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000"} Oct 30 20:02:17.214721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.214553 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" event={"ID":"918edd0c-ea25-4710-8d80-2ee6616501a9","Type":"ContainerStarted","Data":"2c660c16e9111c4bcf4631ae0138151f38a86eb3d474049fd17ceefb61c5f472"} Oct 30 20:02:17.214841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.214819 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:17.262313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:17.262255 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" podStartSLOduration=2.2622405309999998 podStartE2EDuration="2.262240531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:17.258377247 +0000 UTC m=+1604.707869973" watchObservedRunningTime="2026-10-30 20:02:17.262240531 +0000 UTC m=+1604.711733275" Oct 30 20:02:18.047462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:18.047426 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:18.047643 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:18.047484 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:19.047721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:19.047688 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:19.048108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:19.047742 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:20.047068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:20.047033 5991 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:02:20.047255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:20.047087 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 20:02:20.876065 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:20.876016 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:02:21.188760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:21.188041 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:21.190008 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:21.189983 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:21.213483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:21.195739 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:02:21.771533 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:21.771493 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 20:02:21.771790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:21.771761 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" containerID="cri-o://2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502" gracePeriod=30 Oct 30 20:02:27.085152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.085117 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:02:27.190664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.190623 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.190973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.190829 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.190973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.190871 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191179 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191051 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191179 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191097 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191179 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191163 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191255 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191309 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.191481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.191436 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\") pod \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\" (UID: \"5a764cab-894a-4a5d-9aa4-cad98acd9a69\") " Oct 30 20:02:27.192276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192159 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:27.192444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192344 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.192444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192350 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:27.192633 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192581 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:27.192769 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192732 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:27.192856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.192771 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:27.197308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.197168 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:27.197308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.197264 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:27.198255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.198214 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:27.199234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.199209 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4" pod "5a764cab-894a-4a5d-9aa4-cad98acd9a69" (UID: "5a764cab-894a-4a5d-9aa4-cad98acd9a69"). InnerVolumeSpecName "kube-api-access-ww8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:27.293711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293669 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww8x4\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-kube-api-access-ww8x4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293707 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293717 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293725 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293732 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293740 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293747 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a764cab-894a-4a5d-9aa4-cad98acd9a69-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.293949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.293755 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a764cab-894a-4a5d-9aa4-cad98acd9a69-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.313883 5991 generic.go:356] "Generic (PLEG): container finished" podID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerID="2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502" exitCode=0 Oct 30 20:02:27.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.313943 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" Oct 30 20:02:27.314223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.313977 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerDied","Data":"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502"} Oct 30 20:02:27.314223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.314041 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb" event={"ID":"5a764cab-894a-4a5d-9aa4-cad98acd9a69","Type":"ContainerDied","Data":"d1455149a73040bd666046b550fa12552233394da583e840e69cc3eff9a464f8"} Oct 30 20:02:27.314223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.314067 5991 scope.go:122] "RemoveContainer" containerID="2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502" Oct 30 20:02:27.369388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.367715 5991 scope.go:122] "RemoveContainer" containerID="9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a" Oct 30 20:02:27.461388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.456507 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 20:02:27.464568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.464529 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d96913-openshift-default-5cfd95bf99-nchlb"] Oct 30 20:02:27.475809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.475780 5991 scope.go:122] "RemoveContainer" containerID="2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502" Oct 30 20:02:27.477441 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:27.477114 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502\": container with ID starting with 2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502 not found: ID does not exist" containerID="2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502" Oct 30 20:02:27.477674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.477635 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502"} err="failed to get container status \"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502\": rpc error: code = NotFound desc = could not find container \"2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502\": container with ID starting with 2fe3abe67356c2e3b328ae918b1d2c79366fa55e17862a9ffc26754a0e076502 not found: ID does not exist" Oct 30 20:02:27.477851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.477836 5991 scope.go:122] "RemoveContainer" containerID="9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a" Oct 30 20:02:27.478340 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:27.478319 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a\": container with ID starting with 9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a not found: ID does not exist" containerID="9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a" Oct 30 20:02:27.478537 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:27.478511 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a"} err="failed to get container status \"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a\": rpc error: code = NotFound desc = could not find container \"9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a\": container with ID starting with 9d6ac0b5ac2c5eda2e1cac2aa59b7d30b6c4352af48f43b260e16922a7f5260a not found: ID does not exist" Oct 30 20:02:28.225039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:28.225005 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:02:28.681924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:28.681835 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" path="/var/lib/kubelet/pods/5a764cab-894a-4a5d-9aa4-cad98acd9a69/volumes" Oct 30 20:02:31.697383 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:31.697123 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 20:02:31.697840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:31.697400 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" containerID="cri-o://f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4" gracePeriod=30 Oct 30 20:02:37.083879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.083836 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 20:02:37.104666 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.104630 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.104867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.104755 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.104867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.104800 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.104867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.104837 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.104879 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105022 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105137 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105052 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105155 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105177 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:37.105276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105204 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\") pod \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\" (UID: \"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2\") " Oct 30 20:02:37.105276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105255 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:37.105619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105566 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:37.105619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105568 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:37.105822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.105623 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:37.106100 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.106077 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.106192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.106102 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.106192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.106114 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.106192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.106124 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.106192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.106135 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.107296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.107267 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:37.107467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.107289 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:37.107529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.107473 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:37.107529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.107501 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658" pod "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" (UID: "3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2"). InnerVolumeSpecName "kube-api-access-4k658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:37.211106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.211067 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.211106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.211107 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.211427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.211124 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k658\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-kube-api-access-4k658\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.211427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.211138 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.495942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.495899 5991 generic.go:356] "Generic (PLEG): container finished" podID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerID="f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4" exitCode=0 Oct 30 20:02:37.496117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.495938 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerDied","Data":"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4"} Oct 30 20:02:37.496117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.495979 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" Oct 30 20:02:37.496117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.496021 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx" event={"ID":"3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2","Type":"ContainerDied","Data":"ba16499c1e553f6adf381f8d39a1552bba47f0d7c3d53b581561f48bfbaad7c6"} Oct 30 20:02:37.496117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.496044 5991 scope.go:122] "RemoveContainer" containerID="f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4" Oct 30 20:02:37.507804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.507783 5991 scope.go:122] "RemoveContainer" containerID="125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1" Oct 30 20:02:37.518816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.518795 5991 scope.go:122] "RemoveContainer" containerID="f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4" Oct 30 20:02:37.519095 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:37.519073 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4\": container with ID starting with f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4 not found: ID does not exist" containerID="f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4" Oct 30 20:02:37.519174 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.519109 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4"} err="failed to get container status \"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4\": rpc error: code = NotFound desc = could not find container \"f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4\": container with ID starting with f87921ec11f4d9a9acaee97de3ddd4fc8e06a120da9618a6446fb4c5a132b3a4 not found: ID does not exist" Oct 30 20:02:37.519174 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.519133 5991 scope.go:122] "RemoveContainer" containerID="125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1" Oct 30 20:02:37.519426 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:37.519410 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1\": container with ID starting with 125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1 not found: ID does not exist" containerID="125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1" Oct 30 20:02:37.519474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.519433 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1"} err="failed to get container status \"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1\": rpc error: code = NotFound desc = could not find container \"125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1\": container with ID starting with 125ebec7df7ca9c96bab9b881904fc7327d920960456ea806d36f31cbf29efa1 not found: ID does not exist" Oct 30 20:02:37.553279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.553246 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 20:02:37.557627 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:37.557595 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-540a43-openshift-default-8bc477b49-x8tqx"] Oct 30 20:02:38.683297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.683259 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" path="/var/lib/kubelet/pods/3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2/volumes" Oct 30 20:02:38.683757 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.683672 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:02:38.684499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684483 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.684554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684500 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.684554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684519 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.684554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684525 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.684677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684669 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.684708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684685 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.684708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.684696 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.689032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.689002 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.704187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.704156 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:02:38.704462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.704297 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ebe57b\"" Oct 30 20:02:38.704462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.704296 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ebe57b-dockercfg-mmzpt\"" Oct 30 20:02:38.738051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.738008 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqj5p\" (UniqueName: \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.738273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.738175 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.809463 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.810911 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.810937 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a764cab-894a-4a5d-9aa4-cad98acd9a69" containerName="istio-proxy" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.810965 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.810973 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.811200 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b593a0d-cebc-4ce9-af1b-ccd5c70e8ea2" containerName="istio-proxy" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.841569 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.841924 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqj5p\" (UniqueName: \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.869390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.848269 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.983452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.968052 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqj5p\" (UniqueName: \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\") pod \"payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:38.983452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:38.970306 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.009918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.009885 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:39.011504 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.011474 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:02:39.068658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.068617 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.068843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.068795 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncmb\" (UniqueName: \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.171199 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.170922 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.171199 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.171069 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncmb\" (UniqueName: \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.172737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.172042 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.191862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.191830 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncmb\" (UniqueName: \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\") pod \"payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.309202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.309120 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:39.323912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.323880 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:02:39.324877 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:02:39.324838 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c863987_0379_4189_9e5e_46836126458d.slice/crio-085ddf8da6ff3d57e5366b5a86cbe710e974e2188da0df8b65492391305994dc WatchSource:0}: Error finding container 085ddf8da6ff3d57e5366b5a86cbe710e974e2188da0df8b65492391305994dc: Status 404 returned error can't find the container with id 085ddf8da6ff3d57e5366b5a86cbe710e974e2188da0df8b65492391305994dc Oct 30 20:02:39.517001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.516963 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" event={"ID":"1c863987-0379-4189-9e5e-46836126458d","Type":"ContainerStarted","Data":"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7"} Oct 30 20:02:39.517221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.517025 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" event={"ID":"1c863987-0379-4189-9e5e-46836126458d","Type":"ContainerStarted","Data":"085ddf8da6ff3d57e5366b5a86cbe710e974e2188da0df8b65492391305994dc"} Oct 30 20:02:39.517221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.517097 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:39.560802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.560678 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" podStartSLOduration=1.560656357 podStartE2EDuration="1.560656357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:39.557633705 +0000 UTC m=+1627.007126428" watchObservedRunningTime="2026-10-30 20:02:39.560656357 +0000 UTC m=+1627.010149084" Oct 30 20:02:39.632499 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:02:39.632466 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86c9aa_1614_49a8_8008_95bdd7f2b77b.slice/crio-94525d371ca204b0b58db58d7590389cfbd5c50be367769204aa6350e1064536 WatchSource:0}: Error finding container 94525d371ca204b0b58db58d7590389cfbd5c50be367769204aa6350e1064536: Status 404 returned error can't find the container with id 94525d371ca204b0b58db58d7590389cfbd5c50be367769204aa6350e1064536 Oct 30 20:02:39.632912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:39.632887 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:02:40.528066 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:40.528027 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" event={"ID":"0e86c9aa-1614-49a8-8008-95bdd7f2b77b","Type":"ContainerStarted","Data":"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107"} Oct 30 20:02:40.528564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:40.528091 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" event={"ID":"0e86c9aa-1614-49a8-8008-95bdd7f2b77b","Type":"ContainerStarted","Data":"94525d371ca204b0b58db58d7590389cfbd5c50be367769204aa6350e1064536"} Oct 30 20:02:40.528564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:40.528119 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:40.576519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:40.576462 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" podStartSLOduration=2.576445526 podStartE2EDuration="2.576445526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:40.575173321 +0000 UTC m=+1628.024666040" watchObservedRunningTime="2026-10-30 20:02:40.576445526 +0000 UTC m=+1628.025938248" Oct 30 20:02:44.900722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:44.900623 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 20:02:44.901190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:44.900846 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" containerID="cri-o://c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6" gracePeriod=30 Oct 30 20:02:47.092147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.092111 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:02:47.098574 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.098546 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.121950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.121916 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:02:47.127640 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.127604 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-936f70-serving-cert\"" Oct 30 20:02:47.211187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.211152 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.211399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.211356 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs76m\" (UniqueName: \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.211730 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.211709 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.211868 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.211847 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.313485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.313448 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.313671 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.313551 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.313671 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:47.313602 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-serving-cert: secret "maas-api-e2e-inf-b-936f70-serving-cert" not found Oct 30 20:02:47.313671 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.313642 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs76m\" (UniqueName: \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.313867 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:47.313681 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls podName:c7091269-3876-4331-9bb8-4b2a2c73bf18 nodeName:}" failed. No retries permitted until 2026-10-30 20:02:47.81366212 +0000 UTC m=+1635.263154823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls") pod "maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" (UID: "c7091269-3876-4331-9bb8-4b2a2c73bf18") : secret "maas-api-e2e-inf-b-936f70-serving-cert" not found Oct 30 20:02:47.313935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.313872 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.314303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.314276 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.316711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.316685 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.368691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.368610 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs76m\" (UniqueName: \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.820686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.820642 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:47.823513 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:47.823478 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") pod \"maas-api-e2e-inf-b-936f70-7d8df79f6-k982f\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:48.018306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:48.018267 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:48.364892 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:02:48.364826 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7091269_3876_4331_9bb8_4b2a2c73bf18.slice/crio-ae33a2d13c8e2a6d1d1f5f013f2c066cc519ef2be3c85ea63d379f820f32efdd WatchSource:0}: Error finding container ae33a2d13c8e2a6d1d1f5f013f2c066cc519ef2be3c85ea63d379f820f32efdd: Status 404 returned error can't find the container with id ae33a2d13c8e2a6d1d1f5f013f2c066cc519ef2be3c85ea63d379f820f32efdd Oct 30 20:02:48.366615 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:48.366584 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:02:48.633152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:48.632945 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" event={"ID":"c7091269-3876-4331-9bb8-4b2a2c73bf18","Type":"ContainerStarted","Data":"ae33a2d13c8e2a6d1d1f5f013f2c066cc519ef2be3c85ea63d379f820f32efdd"} Oct 30 20:02:49.645357 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:49.645314 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" event={"ID":"c7091269-3876-4331-9bb8-4b2a2c73bf18","Type":"ContainerStarted","Data":"3355c54095ef2aa2be82370faac5b4736d34c5f83349444928b2d7e85f1f68b4"} Oct 30 20:02:49.645904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:49.645482 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:49.692431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:49.690693 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" podStartSLOduration=2.3198556200000002 podStartE2EDuration="2.690674392s" totalImagesPullingTime="370.818772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:49.69035016 +0000 UTC m=+1637.139842915" watchObservedRunningTime="2026-10-30 20:02:49.690674392 +0000 UTC m=+1637.140167119" Oct 30 20:02:50.251589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.251555 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 20:02:50.401784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.401748 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.401964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.401793 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.401964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.401853 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.401964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.401917 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.401995 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402055 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.402135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402126 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402161 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402168 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.402299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402253 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402353 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.402508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402356 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\") pod \"0bf3d919-899c-4fcb-bbb8-704b70166626\" (UID: \"0bf3d919-899c-4fcb-bbb8-704b70166626\") " Oct 30 20:02:50.402508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402415 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.402721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.402697 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:50.403309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.403283 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.403475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.403312 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.403475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.403336 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf3d919-899c-4fcb-bbb8-704b70166626-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.403475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.403351 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.403475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.403380 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.404463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.404433 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:50.404463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.404452 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "kube-api-access-2fc4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.404585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.404519 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.404585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.404553 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy" pod "0bf3d919-899c-4fcb-bbb8-704b70166626" (UID: "0bf3d919-899c-4fcb-bbb8-704b70166626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.504782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.504743 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.504782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.504778 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fc4x\" (UniqueName: \"kubernetes.io/projected/0bf3d919-899c-4fcb-bbb8-704b70166626-kube-api-access-2fc4x\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.504782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.504790 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.505053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.504801 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf3d919-899c-4fcb-bbb8-704b70166626-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.533097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.533067 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:02:50.656088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.656048 5991 generic.go:356] "Generic (PLEG): container finished" podID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerID="c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6" exitCode=0 Oct 30 20:02:50.656580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.656126 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" Oct 30 20:02:50.656580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.656142 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" event={"ID":"0bf3d919-899c-4fcb-bbb8-704b70166626","Type":"ContainerDied","Data":"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6"} Oct 30 20:02:50.656580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.656202 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j" event={"ID":"0bf3d919-899c-4fcb-bbb8-704b70166626","Type":"ContainerDied","Data":"589b56f20362d3a31c2794396430824d3df1036b97e2b27a03bc0611143bfd95"} Oct 30 20:02:50.656580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.656229 5991 scope.go:122] "RemoveContainer" containerID="c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6" Oct 30 20:02:50.671075 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.670943 5991 scope.go:122] "RemoveContainer" containerID="c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6" Oct 30 20:02:50.671442 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:50.671411 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6\": container with ID starting with c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6 not found: ID does not exist" containerID="c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6" Oct 30 20:02:50.671576 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.671453 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6"} err="failed to get container status \"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6\": rpc error: code = NotFound desc = could not find container \"c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6\": container with ID starting with c66f4ed3df3a0b22b5408fb099fcbe75a16883a97d65131aa5fd5d2d9b35e9c6 not found: ID does not exist" Oct 30 20:02:50.713162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.713105 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 20:02:50.724086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:50.724053 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8qd4j"] Oct 30 20:02:50.874332 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:50.874247 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:02:51.539708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:51.539677 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:02:52.682093 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:52.682056 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" path="/var/lib/kubelet/pods/0bf3d919-899c-4fcb-bbb8-704b70166626/volumes" Oct 30 20:02:55.194113 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.194072 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 20:02:55.194895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.194327 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" podUID="349e784b-9359-45ec-92b2-cc7648e017a7" containerName="maas-api" containerID="cri-o://b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2" gracePeriod=30 Oct 30 20:02:55.488525 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.488497 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 20:02:55.508766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.508735 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\" (UniqueName: \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\") pod \"349e784b-9359-45ec-92b2-cc7648e017a7\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " Oct 30 20:02:55.508941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.508916 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\") pod \"349e784b-9359-45ec-92b2-cc7648e017a7\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " Oct 30 20:02:55.508987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.508950 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\") pod \"349e784b-9359-45ec-92b2-cc7648e017a7\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " Oct 30 20:02:55.508987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.508969 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\") pod \"349e784b-9359-45ec-92b2-cc7648e017a7\" (UID: \"349e784b-9359-45ec-92b2-cc7648e017a7\") " Oct 30 20:02:55.509375 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.509335 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca" pod "349e784b-9359-45ec-92b2-cc7648e017a7" (UID: "349e784b-9359-45ec-92b2-cc7648e017a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:55.509693 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.509673 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349e784b-9359-45ec-92b2-cc7648e017a7-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:55.510980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.510954 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb" pod "349e784b-9359-45ec-92b2-cc7648e017a7" (UID: "349e784b-9359-45ec-92b2-cc7648e017a7"). InnerVolumeSpecName "kube-api-access-6glfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:55.511130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.511110 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls" pod "349e784b-9359-45ec-92b2-cc7648e017a7" (UID: "349e784b-9359-45ec-92b2-cc7648e017a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:55.511204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.511188 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs" pod "349e784b-9359-45ec-92b2-cc7648e017a7" (UID: "349e784b-9359-45ec-92b2-cc7648e017a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:55.611124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.611079 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glfb\" (UniqueName: \"kubernetes.io/projected/349e784b-9359-45ec-92b2-cc7648e017a7-kube-api-access-6glfb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:55.611124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.611115 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:55.611124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.611124 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349e784b-9359-45ec-92b2-cc7648e017a7-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:02:55.664048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.664017 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:02:55.850386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.850288 5991 generic.go:356] "Generic (PLEG): container finished" podID="349e784b-9359-45ec-92b2-cc7648e017a7" containerID="b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2" exitCode=0 Oct 30 20:02:55.850386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.850383 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" event={"ID":"349e784b-9359-45ec-92b2-cc7648e017a7","Type":"ContainerDied","Data":"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2"} Oct 30 20:02:55.850677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.850440 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" event={"ID":"349e784b-9359-45ec-92b2-cc7648e017a7","Type":"ContainerDied","Data":"6b5a8e7986b74474722056082ad247bc58ce9c46e387a279647bbc3d27f0dc10"} Oct 30 20:02:55.850677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.850473 5991 scope.go:122] "RemoveContainer" containerID="b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2" Oct 30 20:02:55.850786 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.850708 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md" Oct 30 20:02:55.886385 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.886194 5991 scope.go:122] "RemoveContainer" containerID="b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2" Oct 30 20:02:55.886837 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:02:55.886805 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2\": container with ID starting with b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2 not found: ID does not exist" containerID="b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2" Oct 30 20:02:55.886950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.886853 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2"} err="failed to get container status \"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2\": rpc error: code = NotFound desc = could not find container \"b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2\": container with ID starting with b0898b9ef8b5c2726d044224d2ad06db40ef80e78b70d2fcdde1652dd2ac0ea2 not found: ID does not exist" Oct 30 20:02:55.918759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.918716 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 20:02:55.923545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:55.923511 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa42c3-5c6b768959-7t7md"] Oct 30 20:02:56.681482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:02:56.681444 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349e784b-9359-45ec-92b2-cc7648e017a7" path="/var/lib/kubelet/pods/349e784b-9359-45ec-92b2-cc7648e017a7/volumes" Oct 30 20:03:04.270347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:04.270304 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 20:03:04.271069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:04.270953 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" containerID="cri-o://f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f" gracePeriod=30 Oct 30 20:03:09.145162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.145122 5991 patch_prober.go:29] interesting pod/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:03:09.145625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.145182 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:03:09.583014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.582985 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:03:09.779279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779238 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779292 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779346 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779398 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779461 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779492 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779520 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779572 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.779896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779631 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:09.779896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779696 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:09.780077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779916 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\") pod \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\" (UID: \"7fd6681a-76e7-4482-8f0c-4ee3386540ab\") " Oct 30 20:03:09.780077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.779955 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:09.780077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.780045 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:09.780292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.780255 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:09.781306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781275 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.781511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781487 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.781616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781513 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.781616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781527 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.781616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781543 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.781883 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.781852 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "kube-api-access-njg5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:09.782155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.782127 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:09.782155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.782143 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:09.782278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.782170 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token" pod "7fd6681a-76e7-4482-8f0c-4ee3386540ab" (UID: "7fd6681a-76e7-4482-8f0c-4ee3386540ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:09.882806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.882765 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.882806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.882803 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.882806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.882812 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.883051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.882821 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njg5h\" (UniqueName: \"kubernetes.io/projected/7fd6681a-76e7-4482-8f0c-4ee3386540ab-kube-api-access-njg5h\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:09.977990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.977948 5991 generic.go:356] "Generic (PLEG): container finished" podID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerID="f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f" exitCode=0 Oct 30 20:03:09.978184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.978012 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" Oct 30 20:03:09.978184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.978036 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerDied","Data":"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f"} Oct 30 20:03:09.978184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.978102 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg" event={"ID":"7fd6681a-76e7-4482-8f0c-4ee3386540ab","Type":"ContainerDied","Data":"0bf2fc86eacd5c02cf47f1cdea96b96b3706f0d2ac83adca99d82ce90fdc59bf"} Oct 30 20:03:09.978184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.978129 5991 scope.go:122] "RemoveContainer" containerID="f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f" Oct 30 20:03:09.991206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:09.991184 5991 scope.go:122] "RemoveContainer" containerID="00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed" Oct 30 20:03:10.003690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.003663 5991 scope.go:122] "RemoveContainer" containerID="f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f" Oct 30 20:03:10.004085 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:10.004062 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f\": container with ID starting with f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f not found: ID does not exist" containerID="f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f" Oct 30 20:03:10.004181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.004093 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f"} err="failed to get container status \"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f\": rpc error: code = NotFound desc = could not find container \"f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f\": container with ID starting with f45249e3d4be122d5d8648ad46f77658818249ceb49a81257e31abb5b688730f not found: ID does not exist" Oct 30 20:03:10.004181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.004111 5991 scope.go:122] "RemoveContainer" containerID="00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed" Oct 30 20:03:10.004411 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:10.004390 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed\": container with ID starting with 00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed not found: ID does not exist" containerID="00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed" Oct 30 20:03:10.004486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.004417 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed"} err="failed to get container status \"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed\": rpc error: code = NotFound desc = could not find container \"00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed\": container with ID starting with 00dbc20c409ba56675f94905d5a1af767c388098eb5cb4d7fd1bd126525a26ed not found: ID does not exist" Oct 30 20:03:10.031851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.031756 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 20:03:10.045399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.045345 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aa42c3-openshift-default-56788456c7-wqkqg"] Oct 30 20:03:10.282724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.282639 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:03:10.283569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283552 5991 cpu_manager.go:414] "containerMap: removing container" podUID="349e784b-9359-45ec-92b2-cc7648e017a7" containerName="maas-api" Oct 30 20:03:10.283621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283569 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349e784b-9359-45ec-92b2-cc7648e017a7" containerName="maas-api" Oct 30 20:03:10.283621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283590 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" Oct 30 20:03:10.283621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283595 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" Oct 30 20:03:10.283621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283604 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:10.283621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283610 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:10.283775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283657 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:10.283775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283662 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:10.283854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283780 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf3d919-899c-4fcb-bbb8-704b70166626" containerName="istio-proxy" Oct 30 20:03:10.283854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283790 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:10.283854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.283807 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="349e784b-9359-45ec-92b2-cc7648e017a7" containerName="maas-api" Oct 30 20:03:10.289282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.289261 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.306352 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.306313 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c17154-serving-cert\"" Oct 30 20:03:10.311157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.311123 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:03:10.389912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.389870 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.390103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.389931 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24k4\" (UniqueName: \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.390103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.390029 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.390236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.390216 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.492573 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.492536 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.492809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.492790 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.492879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.492860 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24k4\" (UniqueName: \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.492986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.492960 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.493499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.493475 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.495200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.495176 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.495310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.495227 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.512820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.512789 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24k4\" (UniqueName: \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\") pod \"maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.608201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.608120 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:10.686611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.686573 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" path="/var/lib/kubelet/pods/7fd6681a-76e7-4482-8f0c-4ee3386540ab/volumes" Oct 30 20:03:10.933173 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:10.933141 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807ddeb4_5347_4f20_8f10_5e800bd0f2df.slice/crio-cc4738540fe9003cca26d7d9e3a7453247e37dfd210947832fe3f736695bbe7b WatchSource:0}: Error finding container cc4738540fe9003cca26d7d9e3a7453247e37dfd210947832fe3f736695bbe7b: Status 404 returned error can't find the container with id cc4738540fe9003cca26d7d9e3a7453247e37dfd210947832fe3f736695bbe7b Oct 30 20:03:10.934622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.934598 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:03:10.989575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:10.989535 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" event={"ID":"807ddeb4-5347-4f20-8f10-5e800bd0f2df","Type":"ContainerStarted","Data":"cc4738540fe9003cca26d7d9e3a7453247e37dfd210947832fe3f736695bbe7b"} Oct 30 20:03:12.005290 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:12.005238 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" event={"ID":"807ddeb4-5347-4f20-8f10-5e800bd0f2df","Type":"ContainerStarted","Data":"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16"} Oct 30 20:03:12.005754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:12.005331 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:12.053069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:12.053011 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" podStartSLOduration=1.775246563 podStartE2EDuration="2.052993299s" totalImagesPullingTime="277.746736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:12.049693794 +0000 UTC m=+1659.499186519" watchObservedRunningTime="2026-10-30 20:03:12.052993299 +0000 UTC m=+1659.502486024" Oct 30 20:03:13.856441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:13.856401 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:03:13.856957 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:13.856694 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" podUID="4972ae4e-0a8d-44be-9511-1f307748b48c" containerName="maas-api" containerID="cri-o://9149bf379ddb4d9a93c5f88bb7eff28e0f2221e8b686dbf998ea04e207765597" gracePeriod=30 Oct 30 20:03:14.024603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.024569 5991 generic.go:356] "Generic (PLEG): container finished" podID="4972ae4e-0a8d-44be-9511-1f307748b48c" containerID="9149bf379ddb4d9a93c5f88bb7eff28e0f2221e8b686dbf998ea04e207765597" exitCode=0 Oct 30 20:03:14.024788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.024639 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" event={"ID":"4972ae4e-0a8d-44be-9511-1f307748b48c","Type":"ContainerDied","Data":"9149bf379ddb4d9a93c5f88bb7eff28e0f2221e8b686dbf998ea04e207765597"} Oct 30 20:03:14.215900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.215866 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:03:14.333250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.330720 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\" (UniqueName: \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\") pod \"4972ae4e-0a8d-44be-9511-1f307748b48c\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " Oct 30 20:03:14.333250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.330927 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\") pod \"4972ae4e-0a8d-44be-9511-1f307748b48c\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " Oct 30 20:03:14.333250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.331082 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\") pod \"4972ae4e-0a8d-44be-9511-1f307748b48c\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " Oct 30 20:03:14.333250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.331175 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\") pod \"4972ae4e-0a8d-44be-9511-1f307748b48c\" (UID: \"4972ae4e-0a8d-44be-9511-1f307748b48c\") " Oct 30 20:03:14.366387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.355282 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca" pod "4972ae4e-0a8d-44be-9511-1f307748b48c" (UID: "4972ae4e-0a8d-44be-9511-1f307748b48c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:14.405389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.404990 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls" pod "4972ae4e-0a8d-44be-9511-1f307748b48c" (UID: "4972ae4e-0a8d-44be-9511-1f307748b48c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:14.408624 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.408579 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs" pod "4972ae4e-0a8d-44be-9511-1f307748b48c" (UID: "4972ae4e-0a8d-44be-9511-1f307748b48c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:14.416888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.412952 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p" pod "4972ae4e-0a8d-44be-9511-1f307748b48c" (UID: "4972ae4e-0a8d-44be-9511-1f307748b48c"). InnerVolumeSpecName "kube-api-access-6kf9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:14.427768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.427723 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 20:03:14.428096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.428064 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" containerID="cri-o://bef6eab063e7ed941c28d35e4f22dcc182a87335e44591305b27f608319b6966" gracePeriod=30 Oct 30 20:03:14.434807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.434768 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kf9p\" (UniqueName: \"kubernetes.io/projected/4972ae4e-0a8d-44be-9511-1f307748b48c-kube-api-access-6kf9p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.434807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.434806 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4972ae4e-0a8d-44be-9511-1f307748b48c-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.435056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.434853 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.435056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.434869 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4972ae4e-0a8d-44be-9511-1f307748b48c-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.435056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.434898 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 20:03:14.435226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.435124 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" containerID="cri-o://8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71" gracePeriod=30 Oct 30 20:03:14.445820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.441055 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 20:03:14.445820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.441293 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" containerID="cri-o://b8a53cb70f4d184416e3fe1494df7eb4ad9a2f4cd083b7233fe98903b9833256" gracePeriod=30 Oct 30 20:03:14.747266 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.747227 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 20:03:14.747546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:14.747455 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" containerID="cri-o://9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092" gracePeriod=30 Oct 30 20:03:15.034472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.034432 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" event={"ID":"4972ae4e-0a8d-44be-9511-1f307748b48c","Type":"ContainerDied","Data":"88ea35811e446500841dd3c8a45cbe7a835e84f8dca2a4a076eca7bd2e45c4df"} Oct 30 20:03:15.035018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.034506 5991 scope.go:122] "RemoveContainer" containerID="9149bf379ddb4d9a93c5f88bb7eff28e0f2221e8b686dbf998ea04e207765597" Oct 30 20:03:15.035018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.034514 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p" Oct 30 20:03:15.092974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.092925 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:03:15.098125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.098092 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-r5l4p"] Oct 30 20:03:15.111937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.111522 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:03:15.155045 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:15.155002 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:03:16.680606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:16.680574 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4972ae4e-0a8d-44be-9511-1f307748b48c" path="/var/lib/kubelet/pods/4972ae4e-0a8d-44be-9511-1f307748b48c/volumes" Oct 30 20:03:18.020266 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.020234 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:03:18.351646 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.351552 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:03:18.352568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.352543 5991 cpu_manager.go:414] "containerMap: removing container" podUID="4972ae4e-0a8d-44be-9511-1f307748b48c" containerName="maas-api" Oct 30 20:03:18.352568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.352567 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4972ae4e-0a8d-44be-9511-1f307748b48c" containerName="maas-api" Oct 30 20:03:18.352767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.352742 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4972ae4e-0a8d-44be-9511-1f307748b48c" containerName="maas-api" Oct 30 20:03:18.352821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.352774 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd6681a-76e7-4482-8f0c-4ee3386540ab" containerName="istio-proxy" Oct 30 20:03:18.358158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.358136 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.375812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.375778 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ebe57b-serving-cert\"" Oct 30 20:03:18.385886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.385841 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:03:18.419965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.419923 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.420197 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.420091 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.420270 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.420233 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.420339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.420314 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dnl\" (UniqueName: \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.522059 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.521842 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.522059 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.521955 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.522059 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:18.522040 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-serving-cert: secret "maas-api-e2e-inf-a-ebe57b-serving-cert" not found Oct 30 20:03:18.522403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.522144 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.522403 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:18.522168 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls podName:cac1dad7-458c-438e-8b4c-8f910268d663 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:19.022143139 +0000 UTC m=+1666.471635862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls") pod "maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" (UID: "cac1dad7-458c-438e-8b4c-8f910268d663") : secret "maas-api-e2e-inf-a-ebe57b-serving-cert" not found Oct 30 20:03:18.522547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.522405 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dnl\" (UniqueName: \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.525088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.523524 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.530802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.530768 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:18.550518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:18.545325 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dnl\" (UniqueName: \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:19.034068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.034025 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:19.036475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.036438 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:19.278238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.278202 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:19.597073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.597015 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:03:19.628760 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:19.628709 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac1dad7_458c_438e_8b4c_8f910268d663.slice/crio-b09a7b2414aab1325ea08251a38f55cdd27a71353e0e5a507ea331f81ebad98f WatchSource:0}: Error finding container b09a7b2414aab1325ea08251a38f55cdd27a71353e0e5a507ea331f81ebad98f: Status 404 returned error can't find the container with id b09a7b2414aab1325ea08251a38f55cdd27a71353e0e5a507ea331f81ebad98f Oct 30 20:03:19.836145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.836075 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 20:03:19.839171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.839145 5991 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:03:19.839279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.839190 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:03:19.953030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.952996 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953178 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953281 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953338 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:19.953466 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953426 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953527 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953477 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953527 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953518 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953559 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953648 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.953755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953737 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\") pod \"2f923218-9ec5-4f7f-bee7-c303477e64ab\" (UID: \"2f923218-9ec5-4f7f-bee7-c303477e64ab\") " Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.953893 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954089 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954258 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954533 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954556 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954569 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.954616 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954582 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.955072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.954725 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:19.956298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.956272 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:19.957145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.957117 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:19.957975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.957946 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "kube-api-access-jk9rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:19.958071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:19.957982 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token" pod "2f923218-9ec5-4f7f-bee7-c303477e64ab" (UID: "2f923218-9ec5-4f7f-bee7-c303477e64ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.056379 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.056330 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.056379 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.056384 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.056855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.056398 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f923218-9ec5-4f7f-bee7-c303477e64ab-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.056855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.056413 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f923218-9ec5-4f7f-bee7-c303477e64ab-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.056855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.056426 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9rw\" (UniqueName: \"kubernetes.io/projected/2f923218-9ec5-4f7f-bee7-c303477e64ab-kube-api-access-jk9rw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.060758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.060729 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 20:03:20.112475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.112398 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" podUID="3849bbd1-ded0-45b7-88e9-1960561c85fa" containerName="payload-processing" containerID="cri-o://b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c" gracePeriod=25 Oct 30 20:03:20.146536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.146502 5991 generic.go:356] "Generic (PLEG): container finished" podID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerID="bef6eab063e7ed941c28d35e4f22dcc182a87335e44591305b27f608319b6966" exitCode=0 Oct 30 20:03:20.146729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.146670 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" event={"ID":"8a067728-9ddf-48d5-a8f5-04ee189986b3","Type":"ContainerDied","Data":"bef6eab063e7ed941c28d35e4f22dcc182a87335e44591305b27f608319b6966"} Oct 30 20:03:20.149576 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.149544 5991 generic.go:356] "Generic (PLEG): container finished" podID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerID="9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092" exitCode=0 Oct 30 20:03:20.149773 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.149760 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" Oct 30 20:03:20.150190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.150169 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" event={"ID":"22cbf843-85b1-4096-af22-0a05e23a96c6","Type":"ContainerDied","Data":"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092"} Oct 30 20:03:20.150240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.150222 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k" event={"ID":"22cbf843-85b1-4096-af22-0a05e23a96c6","Type":"ContainerDied","Data":"966edfb0698fb867c218444a35b41527fdf6f1f01f057c66756bd4065c2ddf53"} Oct 30 20:03:20.150275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.150252 5991 scope.go:122] "RemoveContainer" containerID="9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092" Oct 30 20:03:20.159331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159280 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" podUID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" containerName="payload-pre-processing" containerID="cri-o://674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced" gracePeriod=25 Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159439 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159664 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159747 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159835 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.159884 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160013 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160348 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160706 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160823 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160850 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.160971 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161012 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\") pod \"22cbf843-85b1-4096-af22-0a05e23a96c6\" (UID: \"22cbf843-85b1-4096-af22-0a05e23a96c6\") " Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161081 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161244 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161861 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161885 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161901 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cbf843-85b1-4096-af22-0a05e23a96c6-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161915 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.161962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.161929 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.165648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.165415 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "kube-api-access-67ssf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.166309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.166266 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:20.166530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.166449 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" event={"ID":"cac1dad7-458c-438e-8b4c-8f910268d663","Type":"ContainerStarted","Data":"b09a7b2414aab1325ea08251a38f55cdd27a71353e0e5a507ea331f81ebad98f"} Oct 30 20:03:20.168077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.168019 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.168872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.168852 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token" pod "22cbf843-85b1-4096-af22-0a05e23a96c6" (UID: "22cbf843-85b1-4096-af22-0a05e23a96c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.169779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.169756 5991 generic.go:356] "Generic (PLEG): container finished" podID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerID="b8a53cb70f4d184416e3fe1494df7eb4ad9a2f4cd083b7233fe98903b9833256" exitCode=0 Oct 30 20:03:20.169885 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.169837 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" event={"ID":"6bbd6c23-a3ad-43d3-a004-65e19d01ba60","Type":"ContainerDied","Data":"b8a53cb70f4d184416e3fe1494df7eb4ad9a2f4cd083b7233fe98903b9833256"} Oct 30 20:03:20.173469 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.173433 5991 generic.go:356] "Generic (PLEG): container finished" podID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerID="8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71" exitCode=0 Oct 30 20:03:20.173630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.173599 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" event={"ID":"2f923218-9ec5-4f7f-bee7-c303477e64ab","Type":"ContainerDied","Data":"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71"} Oct 30 20:03:20.173706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.173668 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" event={"ID":"2f923218-9ec5-4f7f-bee7-c303477e64ab","Type":"ContainerDied","Data":"6a4dab957549be9c601d73e781f21197d1609e085f8d95a5068079f8c83f4bea"} Oct 30 20:03:20.173807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.173793 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k" Oct 30 20:03:20.270302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.270261 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.270302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.270304 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.270548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.270320 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ssf\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-kube-api-access-67ssf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.270548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.270335 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cbf843-85b1-4096-af22-0a05e23a96c6-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.272584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.272548 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 20:03:20.275181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.275151 5991 scope.go:122] "RemoveContainer" containerID="9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092" Oct 30 20:03:20.275704 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:20.275561 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092\": container with ID starting with 9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092 not found: ID does not exist" containerID="9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092" Oct 30 20:03:20.275704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.275598 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092"} err="failed to get container status \"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092\": rpc error: code = NotFound desc = could not find container \"9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092\": container with ID starting with 9bcbd4ff578532de1a8f0ce55de630b6335207748cbc265f330c1caec3658092 not found: ID does not exist" Oct 30 20:03:20.275704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.275623 5991 scope.go:122] "RemoveContainer" containerID="8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71" Oct 30 20:03:20.280198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.280166 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-mnl9k"] Oct 30 20:03:20.292779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.292751 5991 scope.go:122] "RemoveContainer" containerID="8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71" Oct 30 20:03:20.293235 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:20.293207 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71\": container with ID starting with 8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71 not found: ID does not exist" containerID="8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71" Oct 30 20:03:20.293336 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.293247 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71"} err="failed to get container status \"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71\": rpc error: code = NotFound desc = could not find container \"8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71\": container with ID starting with 8d2a4120e207b124481a168e4ecf38e8074a2f0fe70987ee6129c621ebb16b71 not found: ID does not exist" Oct 30 20:03:20.296106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.295758 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 20:03:20.301057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.300950 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 20:03:20.371866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.371769 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.371866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.371842 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.372090 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.371913 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.372319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372289 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.372696 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372674 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.372757 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372729 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.372870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372852 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.372932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372915 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.372996 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.372981 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.373060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373032 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.373112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373072 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.373165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373107 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.373213 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373198 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.373264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373229 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.373417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373282 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.373417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373352 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.373830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373810 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.374655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373853 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.374655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.373956 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\") pod \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\" (UID: \"6bbd6c23-a3ad-43d3-a004-65e19d01ba60\") " Oct 30 20:03:20.374655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.374019 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\") pod \"8a067728-9ddf-48d5-a8f5-04ee189986b3\" (UID: \"8a067728-9ddf-48d5-a8f5-04ee189986b3\") " Oct 30 20:03:20.375008 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.374987 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.375047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.375017 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a067728-9ddf-48d5-a8f5-04ee189986b3-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.375891 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.375835 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.376096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.376015 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.376560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.376477 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.376842 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.376794 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.377502 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.379425 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.379806 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "kube-api-access-czdvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.380096 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.380324 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.380551 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.381666 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.383912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.383682 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.384558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.384520 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.393591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.389908 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "kube-api-access-lnwbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.393591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.390093 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token" pod "6bbd6c23-a3ad-43d3-a004-65e19d01ba60" (UID: "6bbd6c23-a3ad-43d3-a004-65e19d01ba60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.393591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.393007 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo" pod "8a067728-9ddf-48d5-a8f5-04ee189986b3" (UID: "8a067728-9ddf-48d5-a8f5-04ee189986b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:20.476519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476488 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476520 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476538 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdvd\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-kube-api-access-czdvd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476552 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwbd\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-kube-api-access-lnwbd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476568 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476581 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476595 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476607 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476619 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476631 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476644 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476656 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476668 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476680 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476691 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a067728-9ddf-48d5-a8f5-04ee189986b3-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.476729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.476702 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd6c23-a3ad-43d3-a004-65e19d01ba60-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.517040 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.516986 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 20:03:20.524164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.524130 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tvq9k"] Oct 30 20:03:20.536771 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.536739 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:03:20.549549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.549455 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:03:20.583047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.578812 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\" (UniqueName: \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\") pod \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " Oct 30 20:03:20.583047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.578915 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\") pod \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\" (UID: \"7ae0b0c3-283a-413f-aa0b-27b19baf0f18\") " Oct 30 20:03:20.583047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.580303 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume" pod "7ae0b0c3-283a-413f-aa0b-27b19baf0f18" (UID: "7ae0b0c3-283a-413f-aa0b-27b19baf0f18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.588389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.585698 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4" pod "7ae0b0c3-283a-413f-aa0b-27b19baf0f18" (UID: "7ae0b0c3-283a-413f-aa0b-27b19baf0f18"). InnerVolumeSpecName "kube-api-access-h6gv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.681198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.681165 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\") pod \"3849bbd1-ded0-45b7-88e9-1960561c85fa\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " Oct 30 20:03:20.681420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.681243 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\" (UniqueName: \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\") pod \"3849bbd1-ded0-45b7-88e9-1960561c85fa\" (UID: \"3849bbd1-ded0-45b7-88e9-1960561c85fa\") " Oct 30 20:03:20.681711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.681681 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume" pod "3849bbd1-ded0-45b7-88e9-1960561c85fa" (UID: "3849bbd1-ded0-45b7-88e9-1960561c85fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.682343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.682315 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849bbd1-ded0-45b7-88e9-1960561c85fa-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.682475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.682347 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6gv4\" (UniqueName: \"kubernetes.io/projected/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-kube-api-access-h6gv4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.682475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.682376 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae0b0c3-283a-413f-aa0b-27b19baf0f18-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.684319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.684293 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8" pod "3849bbd1-ded0-45b7-88e9-1960561c85fa" (UID: "3849bbd1-ded0-45b7-88e9-1960561c85fa"). InnerVolumeSpecName "kube-api-access-4wfj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.690033 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.690003 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" path="/var/lib/kubelet/pods/22cbf843-85b1-4096-af22-0a05e23a96c6/volumes" Oct 30 20:03:20.690583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.690566 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" path="/var/lib/kubelet/pods/2f923218-9ec5-4f7f-bee7-c303477e64ab/volumes" Oct 30 20:03:20.784274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:20.784245 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfj8\" (UniqueName: \"kubernetes.io/projected/3849bbd1-ded0-45b7-88e9-1960561c85fa-kube-api-access-4wfj8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.872260 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:20.872224 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:03:21.187559 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.187525 5991 generic.go:356] "Generic (PLEG): container finished" podID="3849bbd1-ded0-45b7-88e9-1960561c85fa" containerID="b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c" exitCode=0 Oct 30 20:03:21.188109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.187588 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" Oct 30 20:03:21.188109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.187613 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" event={"ID":"3849bbd1-ded0-45b7-88e9-1960561c85fa","Type":"ContainerDied","Data":"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c"} Oct 30 20:03:21.188109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.187703 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg" event={"ID":"3849bbd1-ded0-45b7-88e9-1960561c85fa","Type":"ContainerDied","Data":"ec670ed078aa354061cfb92e76ed77d7629b41c863c8bc88f1abdd75c2466c66"} Oct 30 20:03:21.188109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.187771 5991 scope.go:122] "RemoveContainer" containerID="b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c" Oct 30 20:03:21.190937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.190407 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" event={"ID":"8a067728-9ddf-48d5-a8f5-04ee189986b3","Type":"ContainerDied","Data":"918fe15f28bf6f8a9c8229e18823c4b44eb15cdd78c51a52e49be43a0d22c8af"} Oct 30 20:03:21.191148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.191132 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc" Oct 30 20:03:21.192784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.192582 5991 generic.go:356] "Generic (PLEG): container finished" podID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" containerID="674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced" exitCode=0 Oct 30 20:03:21.192784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.192738 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" Oct 30 20:03:21.192948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.192849 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" event={"ID":"7ae0b0c3-283a-413f-aa0b-27b19baf0f18","Type":"ContainerDied","Data":"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced"} Oct 30 20:03:21.192948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.192889 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn" event={"ID":"7ae0b0c3-283a-413f-aa0b-27b19baf0f18","Type":"ContainerDied","Data":"d9a83909aef1c5773821b007a0e59bfca4195b269361f69a2cda4a6b33eb2130"} Oct 30 20:03:21.197646 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.197617 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" event={"ID":"cac1dad7-458c-438e-8b4c-8f910268d663","Type":"ContainerStarted","Data":"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7"} Oct 30 20:03:21.197944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.197929 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:21.202170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.201889 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" event={"ID":"6bbd6c23-a3ad-43d3-a004-65e19d01ba60","Type":"ContainerDied","Data":"9f6c361fa2e1b6e8a48b36d5f585c720c20ef64b71439e7699ad75fe64d02655"} Oct 30 20:03:21.202170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.202040 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn" Oct 30 20:03:21.208759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.208732 5991 scope.go:122] "RemoveContainer" containerID="b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c" Oct 30 20:03:21.209075 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:21.209050 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c\": container with ID starting with b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c not found: ID does not exist" containerID="b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c" Oct 30 20:03:21.209173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.209082 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c"} err="failed to get container status \"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c\": rpc error: code = NotFound desc = could not find container \"b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c\": container with ID starting with b34e8a7e0453b33d46ea33f1d75c045d3474ab3e2a30b13f192d0e1fd941383c not found: ID does not exist" Oct 30 20:03:21.209173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.209100 5991 scope.go:122] "RemoveContainer" containerID="bef6eab063e7ed941c28d35e4f22dcc182a87335e44591305b27f608319b6966" Oct 30 20:03:21.220391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.220355 5991 scope.go:122] "RemoveContainer" containerID="674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced" Oct 30 20:03:21.231759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.231727 5991 scope.go:122] "RemoveContainer" containerID="674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced" Oct 30 20:03:21.232111 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:21.232086 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced\": container with ID starting with 674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced not found: ID does not exist" containerID="674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced" Oct 30 20:03:21.232162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.232124 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced"} err="failed to get container status \"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced\": rpc error: code = NotFound desc = could not find container \"674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced\": container with ID starting with 674ce994c86eb3e2ea59da172e13e8bc1b5695c0ff5c1dbcd5b21277a5af1ced not found: ID does not exist" Oct 30 20:03:21.232162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.232153 5991 scope.go:122] "RemoveContainer" containerID="b8a53cb70f4d184416e3fe1494df7eb4ad9a2f4cd083b7233fe98903b9833256" Oct 30 20:03:21.254238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.254173 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" podStartSLOduration=2.785019711 podStartE2EDuration="3.254156963s" totalImagesPullingTime="469.137252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:21.247186053 +0000 UTC m=+1668.696678777" watchObservedRunningTime="2026-10-30 20:03:21.254156963 +0000 UTC m=+1668.703649688" Oct 30 20:03:21.302694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.302653 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 20:03:21.307128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.307098 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-622nc"] Oct 30 20:03:21.328310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.328272 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:03:21.334031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.334000 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-652137-6b8654bdd-j6mqg"] Oct 30 20:03:21.402960 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.402909 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 20:03:21.408804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.408763 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-7hlzn"] Oct 30 20:03:21.442684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.442584 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:03:21.447919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:21.447879 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-652137-5b7596df7f-bq6kn"] Oct 30 20:03:22.681358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:22.681321 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3849bbd1-ded0-45b7-88e9-1960561c85fa" path="/var/lib/kubelet/pods/3849bbd1-ded0-45b7-88e9-1960561c85fa/volumes" Oct 30 20:03:22.682181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:22.682149 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" path="/var/lib/kubelet/pods/6bbd6c23-a3ad-43d3-a004-65e19d01ba60/volumes" Oct 30 20:03:22.682771 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:22.682752 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" path="/var/lib/kubelet/pods/7ae0b0c3-283a-413f-aa0b-27b19baf0f18/volumes" Oct 30 20:03:22.683491 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:22.683469 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" path="/var/lib/kubelet/pods/8a067728-9ddf-48d5-a8f5-04ee189986b3/volumes" Oct 30 20:03:25.576951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.576911 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:03:25.578155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578121 5991 cpu_manager.go:414] "containerMap: removing container" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" Oct 30 20:03:25.578155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578146 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" Oct 30 20:03:25.578155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578162 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3849bbd1-ded0-45b7-88e9-1960561c85fa" containerName="payload-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578170 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3849bbd1-ded0-45b7-88e9-1960561c85fa" containerName="payload-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578190 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" containerName="payload-pre-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578197 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" containerName="payload-pre-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578212 5991 cpu_manager.go:414] "containerMap: removing container" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578219 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578295 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578304 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578314 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578321 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578588 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a067728-9ddf-48d5-a8f5-04ee189986b3" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578607 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f923218-9ec5-4f7f-bee7-c303477e64ab" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578621 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cbf843-85b1-4096-af22-0a05e23a96c6" containerName="istio-proxy" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578632 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3849bbd1-ded0-45b7-88e9-1960561c85fa" containerName="payload-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578644 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae0b0c3-283a-413f-aa0b-27b19baf0f18" containerName="payload-pre-processing" Oct 30 20:03:25.578777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.578656 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbd6c23-a3ad-43d3-a004-65e19d01ba60" containerName="istio-proxy" Oct 30 20:03:25.583256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.583230 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.600898 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.600871 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2224e496-openshift-default-dockercfg-xxkmv\"" Oct 30 20:03:25.620271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.617004 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:03:25.679131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679083 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679218 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679291 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679419 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679456 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zshq\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679552 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679538 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679608 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679578 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679627 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.679766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.679741 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.781958 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782274 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782449 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782425 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782503 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782440 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782503 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782485 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782516 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zshq\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782559 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782597 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782616 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782646 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782695 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.782948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782931 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.783018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.782997 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.783330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.783295 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.784754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.784727 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.785099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.785083 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.802802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.802771 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.803058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.803032 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zshq\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:25.901005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:25.900908 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:26.221482 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:26.221440 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fba16fa_d9b8_4af2_a101_393d0e8c4ae4.slice/crio-70a71351370fb4ebc3475059b506f2555ac7a2c9d02b49983194494a3c670ac1 WatchSource:0}: Error finding container 70a71351370fb4ebc3475059b506f2555ac7a2c9d02b49983194494a3c670ac1: Status 404 returned error can't find the container with id 70a71351370fb4ebc3475059b506f2555ac7a2c9d02b49983194494a3c670ac1 Oct 30 20:03:26.222831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:26.222806 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:03:26.224823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:26.224787 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:26.224964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:26.224861 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:26.224964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:26.224904 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:26.266099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:26.266060 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" event={"ID":"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4","Type":"ContainerStarted","Data":"70a71351370fb4ebc3475059b506f2555ac7a2c9d02b49983194494a3c670ac1"} Oct 30 20:03:27.225614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.225584 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:03:27.278031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.277987 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" event={"ID":"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4","Type":"ContainerStarted","Data":"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2"} Oct 30 20:03:27.329247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.329181 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podStartSLOduration=2.32915954 podStartE2EDuration="2.32915954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.326693519 +0000 UTC m=+1674.776186241" watchObservedRunningTime="2026-10-30 20:03:27.32915954 +0000 UTC m=+1674.778652266" Oct 30 20:03:27.901244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.901203 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:27.903207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.903179 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:27.903347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:27.903229 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:28.901601 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:28.901558 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:28.902001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:28.901619 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:29.901470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:29.901425 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:29.901662 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:29.901489 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:30.902136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:30.902097 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:30.902752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:30.902170 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:31.902142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:31.902107 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:31.902710 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:31.902169 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:32.901847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:32.901809 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:03:32.902071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:32.901875 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:03:33.906920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:33.906892 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:33.907435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:33.907407 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:33.908049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:33.908031 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:03:38.096422 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:38.093111 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:03:40.687812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.687781 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:03:40.695070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.695041 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.720209 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.720170 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:03:40.736530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.736494 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:03:40.746623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.746563 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.769963 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.769927 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770174 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770011 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v999v\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770174 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770116 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770186 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770231 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770266 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770338 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770531 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770468 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.770595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.770530 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.772536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.772505 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:03:40.873327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.872675 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.873553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.873262 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.877139 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877103 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.877330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877160 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.877330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877241 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.877330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877277 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.878887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877329 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.878887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877394 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.878887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.877437 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.878887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.878763 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.878887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.878874 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.878898 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.878979 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.879157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879000 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879066 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879097 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v999v\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.879448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879197 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879238 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kjr\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.879448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879270 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.879977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.879951 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.880331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.880304 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.881061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.880869 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.881061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.880916 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.891553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.891523 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.893530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.893502 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.912617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.912584 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v999v\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.926436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.919882 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.980567 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.980693 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.980734 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kjr\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.980869 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.980947 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.981015 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.981087 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.981114 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.982389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.981189 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.992391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.989460 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.992391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.989727 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.992391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.989935 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.992391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.990469 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:40.992391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:40.990873 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.000912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.000863 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.001671 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.001587 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.012560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.012523 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:41.012737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.012645 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.014078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.014054 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kjr\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.069387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.069129 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:41.341106 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:41.341072 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3c3bb1_3dc4_4a98_ab58_b2b4b74a0a74.slice/crio-a18aefd734a63af788b8e36613e98fedbc5df06a2d7761045e86f931c464c15f WatchSource:0}: Error finding container a18aefd734a63af788b8e36613e98fedbc5df06a2d7761045e86f931c464c15f: Status 404 returned error can't find the container with id a18aefd734a63af788b8e36613e98fedbc5df06a2d7761045e86f931c464c15f Oct 30 20:03:41.341866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.341814 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:03:41.344718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.344678 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.344795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.344770 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.344850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.344812 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.410256 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:41.410220 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdd1581_7b70_405a_86e6_398598387221.slice/crio-439655de09e84b9edb984c2f3ef8e983b506fe54fff89aac64656203c8da5a54 WatchSource:0}: Error finding container 439655de09e84b9edb984c2f3ef8e983b506fe54fff89aac64656203c8da5a54: Status 404 returned error can't find the container with id 439655de09e84b9edb984c2f3ef8e983b506fe54fff89aac64656203c8da5a54 Oct 30 20:03:41.411026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.411002 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:03:41.414323 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.414285 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.414470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.414353 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.414470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.414398 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:41.416410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:41.416379 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" event={"ID":"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74","Type":"ContainerStarted","Data":"a18aefd734a63af788b8e36613e98fedbc5df06a2d7761045e86f931c464c15f"} Oct 30 20:03:42.426790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:42.426750 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" event={"ID":"5bdd1581-7b70-405a-86e6-398598387221","Type":"ContainerStarted","Data":"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b"} Oct 30 20:03:42.427277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:42.426809 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" event={"ID":"5bdd1581-7b70-405a-86e6-398598387221","Type":"ContainerStarted","Data":"439655de09e84b9edb984c2f3ef8e983b506fe54fff89aac64656203c8da5a54"} Oct 30 20:03:42.428873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:42.428844 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" event={"ID":"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74","Type":"ContainerStarted","Data":"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4"} Oct 30 20:03:42.473052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:42.473001 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" podStartSLOduration=2.472986687 podStartE2EDuration="2.472986687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:42.471652188 +0000 UTC m=+1689.921144914" watchObservedRunningTime="2026-10-30 20:03:42.472986687 +0000 UTC m=+1689.922479412" Oct 30 20:03:42.520485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:42.520429 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" podStartSLOduration=2.520415806 podStartE2EDuration="2.520415806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:42.516996291 +0000 UTC m=+1689.966489017" watchObservedRunningTime="2026-10-30 20:03:42.520415806 +0000 UTC m=+1689.969908600" Oct 30 20:03:43.013622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.013585 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:43.020338 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.020313 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:43.070046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.070006 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:43.105662 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.105610 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:43.437279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.437180 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:43.437780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.437388 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:43.438186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.438164 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:03:43.438281 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:43.438212 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:03:50.872097 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:03:50.872057 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:03:58.591710 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.591674 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:03:58.603194 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.603163 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.624654 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.624622 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:03:58.626900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.626871 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0506c9-openshift-default-dockercfg-7sd8j\"" Oct 30 20:03:58.718854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.718811 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719016 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719190 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719311 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719566 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719532 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719593 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719613 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbkd\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719801 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719703 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.719801 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.719750 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.821843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.821794 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.821872 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.821899 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbkd\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.821972 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822010 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822104 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822236 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822304 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822357 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822471 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822649 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822624 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822706 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822854 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.822962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.822927 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.824949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.824920 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.825126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.825109 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.842623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.842556 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbkd\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.842761 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.842667 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:58.928335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:58.928294 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:03:59.248247 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:03:59.248213 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1554151_0309_4745_b68e_7c37f5cde68e.slice/crio-fcbbe976413a1669f3c7028548b41d5c667c3c4bdff0998f66c93ddc1da3e10f WatchSource:0}: Error finding container fcbbe976413a1669f3c7028548b41d5c667c3c4bdff0998f66c93ddc1da3e10f: Status 404 returned error can't find the container with id fcbbe976413a1669f3c7028548b41d5c667c3c4bdff0998f66c93ddc1da3e10f Oct 30 20:03:59.249622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.249593 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:03:59.251215 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.251179 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.251313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.251247 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.251313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.251277 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.631644 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.631565 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" event={"ID":"f1554151-0309-4745-b68e-7c37f5cde68e","Type":"ContainerStarted","Data":"20437a1787977e89748a8e6d27d3ca3a4795846437c03ced9dcca4202966a676"} Oct 30 20:03:59.631644 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.631625 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" event={"ID":"f1554151-0309-4745-b68e-7c37f5cde68e","Type":"ContainerStarted","Data":"fcbbe976413a1669f3c7028548b41d5c667c3c4bdff0998f66c93ddc1da3e10f"} Oct 30 20:03:59.678580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.678526 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podStartSLOduration=1.678510953 podStartE2EDuration="1.678510953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:59.674396654 +0000 UTC m=+1707.123889381" watchObservedRunningTime="2026-10-30 20:03:59.678510953 +0000 UTC m=+1707.128003737" Oct 30 20:03:59.929245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:03:59.929154 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:04:00.026243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:00.026206 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 20:04:00.026469 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:00.026426 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" containerID="cri-o://e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757" gracePeriod=30 Oct 30 20:04:00.931432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:00.931348 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:00.931869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:00.931453 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:01.929112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:01.929070 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:01.929300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:01.929151 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:02.929425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:02.929387 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:02.929903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:02.929464 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:03.929155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:03.929116 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:03.929347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:03.929175 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:04.933387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:04.932390 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:04.933387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:04.932453 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:05.360724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.360699 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 20:04:05.512847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.512529 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.512847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.512599 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.513099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.512884 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.513436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.513409 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.513576 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.513499 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.515069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.514877 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:05.515247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515223 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.515347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515290 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.515743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515696 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.515867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515848 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.515943 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515888 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.516003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.515949 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\") pod \"1bf30b33-c1e5-45ca-aa90-643d7169571e\" (UID: \"1bf30b33-c1e5-45ca-aa90-643d7169571e\") " Oct 30 20:04:05.516790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.516769 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.516887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.516793 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf30b33-c1e5-45ca-aa90-643d7169571e-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.516887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.516807 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.517453 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.517429 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.517590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.517484 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.519028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.519000 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.520800 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.520774 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:05.523083 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.523060 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "kube-api-access-4jn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.523849 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.523827 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token" pod "1bf30b33-c1e5-45ca-aa90-643d7169571e" (UID: "1bf30b33-c1e5-45ca-aa90-643d7169571e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620062 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jn8l\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-kube-api-access-4jn8l\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620099 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620114 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620130 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf30b33-c1e5-45ca-aa90-643d7169571e-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620143 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.620716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.620150 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf30b33-c1e5-45ca-aa90-643d7169571e-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.844441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.843757 5991 generic.go:356] "Generic (PLEG): container finished" podID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerID="e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757" exitCode=0 Oct 30 20:04:05.844441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.843929 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" event={"ID":"1bf30b33-c1e5-45ca-aa90-643d7169571e","Type":"ContainerDied","Data":"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757"} Oct 30 20:04:05.844441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.844060 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" event={"ID":"1bf30b33-c1e5-45ca-aa90-643d7169571e","Type":"ContainerDied","Data":"5c50b714149d12ede7798be78b7188b38e079e4dd216e52125ddbb42b976cbd4"} Oct 30 20:04:05.844441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.844113 5991 scope.go:122] "RemoveContainer" containerID="e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757" Oct 30 20:04:05.844441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.844414 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" Oct 30 20:04:05.923558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.923515 5991 scope.go:122] "RemoveContainer" containerID="e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757" Oct 30 20:04:05.924202 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:05.924179 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757\": container with ID starting with e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757 not found: ID does not exist" containerID="e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757" Oct 30 20:04:05.924435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.924349 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757"} err="failed to get container status \"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757\": rpc error: code = NotFound desc = could not find container \"e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757\": container with ID starting with e631a77488c489de6211e75102539fca0183d78c5ea0a14cf59ac97073a15757 not found: ID does not exist" Oct 30 20:04:05.934289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.931352 5991 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:04:05.934962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.934932 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 20:04:05.955803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.955767 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 20:04:05.967585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:05.967551 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd"] Oct 30 20:04:06.318129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.318091 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:06.318318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.318151 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-c6vdd" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.680830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.680755 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" path="/var/lib/kubelet/pods/1bf30b33-c1e5-45ca-aa90-643d7169571e/volumes" Oct 30 20:04:06.934699 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.934622 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:04:06.935156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.935069 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:04:06.935931 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:06.935909 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:04:14.247294 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.247108 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:04:14.247813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.247463 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" containerID="cri-o://fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2" gracePeriod=30 Oct 30 20:04:14.257153 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.257129 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:04:14.257443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.257419 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" podUID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" containerName="istio-proxy" containerID="cri-o://0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4" gracePeriod=30 Oct 30 20:04:14.264226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.264204 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:04:14.264447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:14.264418 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" podUID="5bdd1581-7b70-405a-86e6-398598387221" containerName="istio-proxy" containerID="cri-o://f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b" gracePeriod=30 Oct 30 20:04:16.022435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.019478 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:04:16.022435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.022060 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" Oct 30 20:04:16.022435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.022082 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" Oct 30 20:04:16.027390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.027347 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf30b33-c1e5-45ca-aa90-643d7169571e" containerName="istio-proxy" Oct 30 20:04:16.047410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.041889 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:04:16.047410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.042014 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.072376 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.072330 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4a31d3-dockercfg-qm8bm\"" Oct 30 20:04:16.072544 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.072344 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4a31d3\"" Oct 30 20:04:16.132233 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.132197 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpwk\" (UniqueName: \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.132442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.132401 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.233760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.233725 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpwk\" (UniqueName: \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.233942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.233830 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.234776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.234750 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.254463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.254433 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpwk\" (UniqueName: \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\") pod \"payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.366661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.366563 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.670122 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:04:16.670085 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8a9ae0_85ce_44a4_b298_40198f94b251.slice/crio-2f91f7869cfd2964f4d5a6ba4e764dd36677bfb8a441603168973651fe8ce9cf WatchSource:0}: Error finding container 2f91f7869cfd2964f4d5a6ba4e764dd36677bfb8a441603168973651fe8ce9cf: Status 404 returned error can't find the container with id 2f91f7869cfd2964f4d5a6ba4e764dd36677bfb8a441603168973651fe8ce9cf Oct 30 20:04:16.682508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.682473 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:04:16.960638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.960548 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" event={"ID":"0d8a9ae0-85ce-44a4-b298-40198f94b251","Type":"ContainerStarted","Data":"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe"} Oct 30 20:04:16.960638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.960621 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:16.960839 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:16.960647 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" event={"ID":"0d8a9ae0-85ce-44a4-b298-40198f94b251","Type":"ContainerStarted","Data":"2f91f7869cfd2964f4d5a6ba4e764dd36677bfb8a441603168973651fe8ce9cf"} Oct 30 20:04:18.909230 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:18.909194 5991 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:04:18.909602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:18.909251 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:04:19.612692 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.612631 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:04:19.649967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.649934 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:04:19.654040 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.654014 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:04:19.671108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.671053 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" podStartSLOduration=4.671035302 podStartE2EDuration="4.671035302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:17.012434279 +0000 UTC m=+1724.461927003" watchObservedRunningTime="2026-10-30 20:04:19.671035302 +0000 UTC m=+1727.120528029" Oct 30 20:04:19.699391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.699339 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.699579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.699431 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.700103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.700075 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.700218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.700180 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.700591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.700322 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.700591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.700382 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.703069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.700422 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.703069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.701117 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.703069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.701238 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\") pod \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\" (UID: \"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74\") " Oct 30 20:04:19.703069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.702904 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.705217 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.704904 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.706805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.706220 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.706805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.706508 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.709306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.708867 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "kube-api-access-v999v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.713414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.709999 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.714554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.713995 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.714554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.714313 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.715643 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.715465 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token" pod "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" (UID: "5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803398 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803488 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803610 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803688 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803725 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803765 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803842 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803900 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803940 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.803966 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804023 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804099 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804142 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804187 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804246 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804330 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\") pod \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\" (UID: \"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804358 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.804433 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\") pod \"5bdd1581-7b70-405a-86e6-398598387221\" (UID: \"5bdd1581-7b70-405a-86e6-398598387221\") " Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805592 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805621 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805636 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805649 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805663 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805676 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805690 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805703 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v999v\" (UniqueName: \"kubernetes.io/projected/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-kube-api-access-v999v\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.805717 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.806043 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.806352 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.806802 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.807120 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.807580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.807455 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.813253 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.812744 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.813253 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.812936 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.813253 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.813083 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.813471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.813266 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.813471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.813458 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.815182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.815127 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "kube-api-access-9zshq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.815418 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.815388 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.815776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.815741 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.817092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.817053 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.817756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.817680 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy" pod "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" (UID: "2fba16fa-d9b8-4af2-a101-393d0e8c4ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.817756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.817683 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.817870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.817825 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.819453 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.819426 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr" pod "5bdd1581-7b70-405a-86e6-398598387221" (UID: "5bdd1581-7b70-405a-86e6-398598387221"). InnerVolumeSpecName "kube-api-access-x5kjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.906980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.906942 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.906980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.906977 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.906980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.906987 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bdd1581-7b70-405a-86e6-398598387221-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.906996 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907005 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907014 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907021 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907031 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bdd1581-7b70-405a-86e6-398598387221-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907038 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907045 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kjr\" (UniqueName: \"kubernetes.io/projected/5bdd1581-7b70-405a-86e6-398598387221-kube-api-access-x5kjr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907056 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907063 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907071 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907080 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zshq\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-kube-api-access-9zshq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907088 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907095 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bdd1581-7b70-405a-86e6-398598387221-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907103 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.907214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.907110 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.996540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.993471 5991 generic.go:356] "Generic (PLEG): container finished" podID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerID="fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2" exitCode=0 Oct 30 20:04:19.996540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.993643 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" event={"ID":"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4","Type":"ContainerDied","Data":"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2"} Oct 30 20:04:19.996540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.993700 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" event={"ID":"2fba16fa-d9b8-4af2-a101-393d0e8c4ae4","Type":"ContainerDied","Data":"70a71351370fb4ebc3475059b506f2555ac7a2c9d02b49983194494a3c670ac1"} Oct 30 20:04:19.996540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.993731 5991 scope.go:122] "RemoveContainer" containerID="fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2" Oct 30 20:04:19.996540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:19.993928 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs" Oct 30 20:04:20.005788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.005755 5991 generic.go:356] "Generic (PLEG): container finished" podID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" containerID="0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4" exitCode=0 Oct 30 20:04:20.005992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.005804 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" Oct 30 20:04:20.006424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.005821 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" event={"ID":"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74","Type":"ContainerDied","Data":"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4"} Oct 30 20:04:20.006689 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.006664 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr" event={"ID":"5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74","Type":"ContainerDied","Data":"a18aefd734a63af788b8e36613e98fedbc5df06a2d7761045e86f931c464c15f"} Oct 30 20:04:20.011602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.009679 5991 generic.go:356] "Generic (PLEG): container finished" podID="5bdd1581-7b70-405a-86e6-398598387221" containerID="f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b" exitCode=0 Oct 30 20:04:20.011602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.009756 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" event={"ID":"5bdd1581-7b70-405a-86e6-398598387221","Type":"ContainerDied","Data":"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b"} Oct 30 20:04:20.011602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.009794 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" event={"ID":"5bdd1581-7b70-405a-86e6-398598387221","Type":"ContainerDied","Data":"439655de09e84b9edb984c2f3ef8e983b506fe54fff89aac64656203c8da5a54"} Oct 30 20:04:20.011602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.009864 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb" Oct 30 20:04:20.029539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.029515 5991 scope.go:122] "RemoveContainer" containerID="fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2" Oct 30 20:04:20.030170 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:20.030149 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2\": container with ID starting with fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2 not found: ID does not exist" containerID="fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2" Oct 30 20:04:20.030260 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.030177 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2"} err="failed to get container status \"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2\": rpc error: code = NotFound desc = could not find container \"fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2\": container with ID starting with fd49f1737a3c69edb60dd7110d260b557211a78f31c69657542b4b6007c96af2 not found: ID does not exist" Oct 30 20:04:20.030260 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.030197 5991 scope.go:122] "RemoveContainer" containerID="0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4" Oct 30 20:04:20.042886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.042861 5991 scope.go:122] "RemoveContainer" containerID="0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4" Oct 30 20:04:20.043162 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:20.043139 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4\": container with ID starting with 0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4 not found: ID does not exist" containerID="0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4" Oct 30 20:04:20.043219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.043172 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4"} err="failed to get container status \"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4\": rpc error: code = NotFound desc = could not find container \"0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4\": container with ID starting with 0c94a060555bfd654151baa361384291727aa04164a465d859e68d33443f83a4 not found: ID does not exist" Oct 30 20:04:20.043219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.043190 5991 scope.go:122] "RemoveContainer" containerID="f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b" Oct 30 20:04:20.054128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.054098 5991 scope.go:122] "RemoveContainer" containerID="f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b" Oct 30 20:04:20.054479 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:20.054456 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b\": container with ID starting with f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b not found: ID does not exist" containerID="f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b" Oct 30 20:04:20.054588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.054490 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b"} err="failed to get container status \"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b\": rpc error: code = NotFound desc = could not find container \"f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b\": container with ID starting with f62f6fc802739298d9e4d68d2aad67d81446f2cbdef48a8eac56374079544b6b not found: ID does not exist" Oct 30 20:04:20.058292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.058224 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:04:20.061958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.061937 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-fhljs"] Oct 30 20:04:20.125190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.125154 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:04:20.128825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.128802 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-8prqr"] Oct 30 20:04:20.170851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.170819 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:04:20.176657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.176627 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-jmzsb"] Oct 30 20:04:20.682200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.682167 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" path="/var/lib/kubelet/pods/2fba16fa-d9b8-4af2-a101-393d0e8c4ae4/volumes" Oct 30 20:04:20.682768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.682752 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" path="/var/lib/kubelet/pods/5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74/volumes" Oct 30 20:04:20.683211 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:20.683199 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdd1581-7b70-405a-86e6-398598387221" path="/var/lib/kubelet/pods/5bdd1581-7b70-405a-86e6-398598387221/volumes" Oct 30 20:04:20.870828 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:20.870793 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:04:27.971829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:27.971798 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:04:33.903110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:33.903076 5991 scope.go:122] "RemoveContainer" containerID="46d948fac3df18530905239ec04732ed08781671515b0c2e76ccc7452e5c7b16" Oct 30 20:04:34.011566 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.011534 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:04:34.011804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.011778 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" podUID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" containerName="maas-api" containerID="cri-o://1857832849bb703c515e859dc0bcdef088a8bc2a83e5ef12b19a8f8de0f2c85f" gracePeriod=30 Oct 30 20:04:34.130728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.130672 5991 generic.go:356] "Generic (PLEG): container finished" podID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" containerID="1857832849bb703c515e859dc0bcdef088a8bc2a83e5ef12b19a8f8de0f2c85f" exitCode=0 Oct 30 20:04:34.130728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.130698 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" event={"ID":"f76498a3-61c2-4e27-aa37-a19caa4a2f83","Type":"ContainerDied","Data":"1857832849bb703c515e859dc0bcdef088a8bc2a83e5ef12b19a8f8de0f2c85f"} Oct 30 20:04:34.365564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.364459 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.471761 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") pod \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.471864 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\" (UniqueName: \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\") pod \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.472028 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\") pod \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.472259 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\") pod \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\" (UID: \"f76498a3-61c2-4e27-aa37-a19caa4a2f83\") " Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.474460 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca" pod "f76498a3-61c2-4e27-aa37-a19caa4a2f83" (UID: "f76498a3-61c2-4e27-aa37-a19caa4a2f83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.486484 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls" pod "f76498a3-61c2-4e27-aa37-a19caa4a2f83" (UID: "f76498a3-61c2-4e27-aa37-a19caa4a2f83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.486869 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs" pod "f76498a3-61c2-4e27-aa37-a19caa4a2f83" (UID: "f76498a3-61c2-4e27-aa37-a19caa4a2f83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:34.489445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.488484 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2" pod "f76498a3-61c2-4e27-aa37-a19caa4a2f83" (UID: "f76498a3-61c2-4e27-aa37-a19caa4a2f83"). InnerVolumeSpecName "kube-api-access-ssvr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:34.574972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.574933 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvr2\" (UniqueName: \"kubernetes.io/projected/f76498a3-61c2-4e27-aa37-a19caa4a2f83-kube-api-access-ssvr2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.574972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.574971 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76498a3-61c2-4e27-aa37-a19caa4a2f83-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.575254 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.574992 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.575254 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:34.575012 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76498a3-61c2-4e27-aa37-a19caa4a2f83-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:35.140046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:35.140019 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" Oct 30 20:04:35.140499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:35.140059 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz" event={"ID":"f76498a3-61c2-4e27-aa37-a19caa4a2f83","Type":"ContainerDied","Data":"bc171557c4c5d39a1db40eedaa4f676524209567b19eccde65126871bb9e5917"} Oct 30 20:04:35.140499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:35.140120 5991 scope.go:122] "RemoveContainer" containerID="1857832849bb703c515e859dc0bcdef088a8bc2a83e5ef12b19a8f8de0f2c85f" Oct 30 20:04:35.197822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:35.197784 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:04:35.208756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:35.208724 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-652137-56bc556b9b-nnzfz"] Oct 30 20:04:36.680997 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:36.680959 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" path="/var/lib/kubelet/pods/f76498a3-61c2-4e27-aa37-a19caa4a2f83/volumes" Oct 30 20:04:39.269183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.269148 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:04:39.270101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270080 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5bdd1581-7b70-405a-86e6-398598387221" containerName="istio-proxy" Oct 30 20:04:39.270148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270103 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdd1581-7b70-405a-86e6-398598387221" containerName="istio-proxy" Oct 30 20:04:39.270148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270125 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" containerName="istio-proxy" Oct 30 20:04:39.270148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270131 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" containerName="istio-proxy" Oct 30 20:04:39.270250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270164 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" containerName="maas-api" Oct 30 20:04:39.270250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270170 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" containerName="maas-api" Oct 30 20:04:39.270250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270185 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" Oct 30 20:04:39.270250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270190 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" Oct 30 20:04:39.270390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270302 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76498a3-61c2-4e27-aa37-a19caa4a2f83" containerName="maas-api" Oct 30 20:04:39.270390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270314 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fba16fa-d9b8-4af2-a101-393d0e8c4ae4" containerName="istio-proxy" Oct 30 20:04:39.270390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270323 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdd1581-7b70-405a-86e6-398598387221" containerName="istio-proxy" Oct 30 20:04:39.270390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.270331 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3c3bb1-3dc4-4a98-ab58-b2b4b74a0a74" containerName="istio-proxy" Oct 30 20:04:39.275211 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.275191 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.290208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.290173 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-58d72c-serving-cert\"" Oct 30 20:04:39.293148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.293123 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:04:39.455865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.455827 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.456079 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.456013 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.456079 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.456044 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2w9\" (UniqueName: \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.456201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.456087 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.557781 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.557677 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.557974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.557856 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.557974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.557888 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2w9\" (UniqueName: \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.557974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.557929 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.559052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.559018 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.561677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.561512 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.561677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.561633 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.579588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.579553 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2w9\" (UniqueName: \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\") pod \"maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.591127 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.591056 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:39.910776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.910743 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:04:39.910938 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:04:39.910772 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdcd795_7d67_4491_bf73_1e739ffac88a.slice/crio-74c85de4c5c375d64142c03546e116aecab2ce2db43c0cf55df811f76dfaa9d5 WatchSource:0}: Error finding container 74c85de4c5c375d64142c03546e116aecab2ce2db43c0cf55df811f76dfaa9d5: Status 404 returned error can't find the container with id 74c85de4c5c375d64142c03546e116aecab2ce2db43c0cf55df811f76dfaa9d5 Oct 30 20:04:39.912600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:39.912579 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:04:40.182029 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:40.181945 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" event={"ID":"3bdcd795-7d67-4491-bf73-1e739ffac88a","Type":"ContainerStarted","Data":"74c85de4c5c375d64142c03546e116aecab2ce2db43c0cf55df811f76dfaa9d5"} Oct 30 20:04:41.198416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:41.193087 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" event={"ID":"3bdcd795-7d67-4491-bf73-1e739ffac88a","Type":"ContainerStarted","Data":"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a"} Oct 30 20:04:41.198416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:41.193197 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:41.239934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:41.239867 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" podStartSLOduration=1.926895161 podStartE2EDuration="2.239850608s" totalImagesPullingTime="312.955447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:41.235954429 +0000 UTC m=+1748.685447156" watchObservedRunningTime="2026-10-30 20:04:41.239850608 +0000 UTC m=+1748.689343414" Oct 30 20:04:42.388442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:42.388405 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:04:42.388926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:42.388671 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" containerID="cri-o://fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043" gracePeriod=30 Oct 30 20:04:47.210675 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.210643 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:04:47.717774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.717740 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:04:47.823723 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.823623 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.823974 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824097 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824183 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824282 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824328 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824461 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824506 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824560 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:47.824719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824658 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\") pod \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\" (UID: \"50cb8bc2-1960-442e-a13c-5c8b2f895bc3\") " Oct 30 20:04:47.824820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824799 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:47.825014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.824992 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:47.825579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.825538 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:47.826054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.825692 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.826054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.825750 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.826054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.825768 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.826054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.825701 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:47.827565 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.827542 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:47.827729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.827704 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "kube-api-access-swtwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:47.827833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.827816 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:47.827993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.827974 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo" pod "50cb8bc2-1960-442e-a13c-5c8b2f895bc3" (UID: "50cb8bc2-1960-442e-a13c-5c8b2f895bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:47.927328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927276 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.927328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927326 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.927328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927338 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtwb\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-kube-api-access-swtwb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.927622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927351 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.927622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927408 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:47.927622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:47.927420 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb8bc2-1960-442e-a13c-5c8b2f895bc3-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:04:48.254709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.254673 5991 generic.go:356] "Generic (PLEG): container finished" podID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerID="fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043" exitCode=0 Oct 30 20:04:48.255171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.254722 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" Oct 30 20:04:48.255171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.254756 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" event={"ID":"50cb8bc2-1960-442e-a13c-5c8b2f895bc3","Type":"ContainerDied","Data":"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043"} Oct 30 20:04:48.255171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.254816 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z" event={"ID":"50cb8bc2-1960-442e-a13c-5c8b2f895bc3","Type":"ContainerDied","Data":"bb191f2063e7d22fdc5b4f36e5390998d1da729dca2bd8205b1b219b4aa0ca1f"} Oct 30 20:04:48.255171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.254841 5991 scope.go:122] "RemoveContainer" containerID="fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043" Oct 30 20:04:48.267819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.267796 5991 scope.go:122] "RemoveContainer" containerID="fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043" Oct 30 20:04:48.268150 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:48.268127 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043\": container with ID starting with fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043 not found: ID does not exist" containerID="fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043" Oct 30 20:04:48.268202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.268160 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043"} err="failed to get container status \"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043\": rpc error: code = NotFound desc = could not find container \"fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043\": container with ID starting with fb9323c7f7801dd6749ae22a4bc2af4e2e0d3d19e8ae6240c20851a85f293043 not found: ID does not exist" Oct 30 20:04:48.312650 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.312617 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:04:48.324827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.324794 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-wqk2z"] Oct 30 20:04:48.682048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:48.681967 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" path="/var/lib/kubelet/pods/50cb8bc2-1960-442e-a13c-5c8b2f895bc3/volumes" Oct 30 20:04:49.896231 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.896035 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:04:49.909403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.897257 5991 cpu_manager.go:414] "containerMap: removing container" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" Oct 30 20:04:49.909403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.897285 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" Oct 30 20:04:49.909403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.897552 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cb8bc2-1960-442e-a13c-5c8b2f895bc3" containerName="istio-proxy" Oct 30 20:04:49.909403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.906470 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:49.939158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.936696 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:04:49.939158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.938016 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c8fc40-dockercfg-c4bs9\"" Oct 30 20:04:49.939158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.938112 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c8fc40\"" Oct 30 20:04:49.996988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.996941 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:49.997161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:49.997097 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmbp\" (UniqueName: \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.098320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.098276 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.098551 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.098342 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmbp\" (UniqueName: \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.099339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.099312 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.118720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.118680 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmbp\" (UniqueName: \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\") pod \"payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.257981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.257937 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:50.568976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:50.568924 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:04:50.569117 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:04:50.569067 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b33c1_a470_485d_b2da_b818ae2f9ca5.slice/crio-e603e203b777d28bc82e5121812222f6bac1df4ab13cd37602ab8c8e7f07440b WatchSource:0}: Error finding container e603e203b777d28bc82e5121812222f6bac1df4ab13cd37602ab8c8e7f07440b: Status 404 returned error can't find the container with id e603e203b777d28bc82e5121812222f6bac1df4ab13cd37602ab8c8e7f07440b Oct 30 20:04:50.871164 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:04:50.871074 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:04:51.282544 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:51.282504 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" event={"ID":"6f4b33c1-a470-485d-b2da-b818ae2f9ca5","Type":"ContainerStarted","Data":"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872"} Oct 30 20:04:51.282931 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:51.282559 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" event={"ID":"6f4b33c1-a470-485d-b2da-b818ae2f9ca5","Type":"ContainerStarted","Data":"e603e203b777d28bc82e5121812222f6bac1df4ab13cd37602ab8c8e7f07440b"} Oct 30 20:04:51.282931 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:51.282602 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:04:51.325226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:04:51.325165 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" podStartSLOduration=2.32514936 podStartE2EDuration="2.32514936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:51.32181452 +0000 UTC m=+1758.771307245" watchObservedRunningTime="2026-10-30 20:04:51.32514936 +0000 UTC m=+1758.774642099" Oct 30 20:05:02.294909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:02.294880 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:05:11.784184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:11.784141 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:05:20.872639 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:05:20.872602 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:05:33.020157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:33.020122 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:05:33.024717 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:33.024692 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:05:34.513557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.513524 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:05:34.519915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.519887 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.547424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.547388 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c8fc40-serving-cert\"" Oct 30 20:05:34.556285 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.556242 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:05:34.614748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.614706 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.614944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.614836 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.614944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.614861 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.615070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.615054 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2vx\" (UniqueName: \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.716515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.716476 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2vx\" (UniqueName: \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.716734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.716579 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.716734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.716640 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.716871 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.716751 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.717438 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.717349 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.719164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.719141 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.719270 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.719184 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.750431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.750345 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2vx\" (UniqueName: \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\") pod \"maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:34.835927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:34.835836 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:35.157740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:35.157702 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:05:35.727226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:35.727181 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" event={"ID":"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c","Type":"ContainerStarted","Data":"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73"} Oct 30 20:05:35.727744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:35.727274 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" event={"ID":"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c","Type":"ContainerStarted","Data":"40436eead5c2adf603acdeb3ea5d9251b1f30b4c359d7ce9b261764c78b26cc4"} Oct 30 20:05:35.727744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:35.727327 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:35.788595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:35.777339 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" podStartSLOduration=1.416567151 podStartE2EDuration="1.777312357s" totalImagesPullingTime="360.745206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:35.772456479 +0000 UTC m=+1803.221949408" watchObservedRunningTime="2026-10-30 20:05:35.777312357 +0000 UTC m=+1803.226805076" Oct 30 20:05:41.742324 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:05:41.742294 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:05:50.876241 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:05:50.876064 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:06:20.667925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:20.667887 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:06:20.872613 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:06:20.872572 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:06:37.864480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:37.864440 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:06:37.869443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:37.869419 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:37.885550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:37.885520 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d3c508\"" Oct 30 20:06:37.885719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:37.885558 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d3c508-dockercfg-tdhj4\"" Oct 30 20:06:37.902470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:37.902432 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:06:38.061512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.060211 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhf7w\" (UniqueName: \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.061512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.060412 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.082581 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.079334 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:06:38.087739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.086696 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:06:38.087739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.086817 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.162137 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.162037 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhf7w\" (UniqueName: \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.162300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.162161 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.162908 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.162884 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.183876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.183845 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhf7w\" (UniqueName: \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\") pod \"payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.184665 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.184645 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:38.264937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.264887 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.265348 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.265318 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dbl\" (UniqueName: \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.366837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.366796 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dbl\" (UniqueName: \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.367060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.366848 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.367682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.367655 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.387154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.387116 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dbl\" (UniqueName: \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\") pod \"payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.411111 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.411069 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:38.510003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.509966 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:06:38.516820 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:06:38.516790 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda5ffeb_27fc_4726_ac2c_b2868ee1202b.slice/crio-58e4530cba9b0387dfc69f3447169149b114ba2b217b9dbb323473a34a4ba14f WatchSource:0}: Error finding container 58e4530cba9b0387dfc69f3447169149b114ba2b217b9dbb323473a34a4ba14f: Status 404 returned error can't find the container with id 58e4530cba9b0387dfc69f3447169149b114ba2b217b9dbb323473a34a4ba14f Oct 30 20:06:38.735959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:38.735847 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:06:38.737655 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:06:38.737624 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c02f4a_63b5_4bb9_abd2_3bda6ce7fbc1.slice/crio-88f54327c6603d2b05c0704e7af318a6eaa9b2ca7cf842294c9bc5250ebdd359 WatchSource:0}: Error finding container 88f54327c6603d2b05c0704e7af318a6eaa9b2ca7cf842294c9bc5250ebdd359: Status 404 returned error can't find the container with id 88f54327c6603d2b05c0704e7af318a6eaa9b2ca7cf842294c9bc5250ebdd359 Oct 30 20:06:39.288898 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.288859 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" event={"ID":"dda5ffeb-27fc-4726-ac2c-b2868ee1202b","Type":"ContainerStarted","Data":"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c"} Oct 30 20:06:39.289451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.288930 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:39.289451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.288954 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" event={"ID":"dda5ffeb-27fc-4726-ac2c-b2868ee1202b","Type":"ContainerStarted","Data":"58e4530cba9b0387dfc69f3447169149b114ba2b217b9dbb323473a34a4ba14f"} Oct 30 20:06:39.290725 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.290698 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" event={"ID":"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1","Type":"ContainerStarted","Data":"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0"} Oct 30 20:06:39.290843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.290744 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" event={"ID":"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1","Type":"ContainerStarted","Data":"88f54327c6603d2b05c0704e7af318a6eaa9b2ca7cf842294c9bc5250ebdd359"} Oct 30 20:06:39.290843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.290770 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:39.329874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.329816 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" podStartSLOduration=2.329800695 podStartE2EDuration="2.329800695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:39.328419789 +0000 UTC m=+1866.777912519" watchObservedRunningTime="2026-10-30 20:06:39.329800695 +0000 UTC m=+1866.779293419" Oct 30 20:06:39.371271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:39.371216 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" podStartSLOduration=2.371199716 podStartE2EDuration="2.371199716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:39.369635498 +0000 UTC m=+1866.819128223" watchObservedRunningTime="2026-10-30 20:06:39.371199716 +0000 UTC m=+1866.820692440" Oct 30 20:06:42.111938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.111898 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:06:42.117543 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.117511 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.136775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.136744 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.136978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.136895 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.136978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.136939 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137086 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137138 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137261 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137309 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5mt\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137466 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137422 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.137526 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.137465 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.145076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.145008 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eb16ce-openshift-default-dockercfg-x2mq2\"" Oct 30 20:06:42.151942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.151901 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:06:42.239397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239335 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239471 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239513 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239589 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239627 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239708 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239744 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5mt\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.239803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239782 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.239818 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240211 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.240187 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240380 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.240298 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.240411 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240647 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.240619 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.240780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.240718 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.242497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.242470 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.242736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.242711 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.264187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.264145 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.264340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.264316 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5mt\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.434045 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.433947 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:42.761837 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:06:42.761788 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af73142_cdcb_45b6_997f_b5f23d759b7d.slice/crio-74e417c8a3b9f905b1e98f1149014f22d536927099eb48420ce9bf6fa1413a99 WatchSource:0}: Error finding container 74e417c8a3b9f905b1e98f1149014f22d536927099eb48420ce9bf6fa1413a99: Status 404 returned error can't find the container with id 74e417c8a3b9f905b1e98f1149014f22d536927099eb48420ce9bf6fa1413a99 Oct 30 20:06:42.763599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.763575 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:06:42.765675 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.765638 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:42.765791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.765714 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:42.765791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:42.765744 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:43.330147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.330103 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" event={"ID":"3af73142-cdcb-45b6-997f-b5f23d759b7d","Type":"ContainerStarted","Data":"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190"} Oct 30 20:06:43.330667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.330170 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" event={"ID":"3af73142-cdcb-45b6-997f-b5f23d759b7d","Type":"ContainerStarted","Data":"74e417c8a3b9f905b1e98f1149014f22d536927099eb48420ce9bf6fa1413a99"} Oct 30 20:06:43.386681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.386624 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podStartSLOduration=1.386607436 podStartE2EDuration="1.386607436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:43.383621425 +0000 UTC m=+1870.833114150" watchObservedRunningTime="2026-10-30 20:06:43.386607436 +0000 UTC m=+1870.836100234" Oct 30 20:06:43.434897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.434853 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:43.436749 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.436719 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:43.436866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:43.436781 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:44.435147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:44.435110 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:44.435607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:44.435166 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:45.434702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:45.434662 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:45.434893 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:45.434722 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:46.434915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:46.434872 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:46.435335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:46.434932 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:47.137730 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.137688 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:06:47.143243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.143214 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.195398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.179206 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:06:47.201307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.201273 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a64be2-dockercfg-9z7h2\"" Oct 30 20:06:47.201689 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.201662 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a64be2\"" Oct 30 20:06:47.246763 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.246698 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dvj\" (UniqueName: \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.247120 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.247101 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.397495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.359610 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dvj\" (UniqueName: \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.397495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.359716 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.397495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.360918 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.450387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.443168 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:47.450387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.443242 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:47.464790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.457473 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dvj\" (UniqueName: \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\") pod \"payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.482992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.482948 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:47.849173 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:06:47.849124 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ab3c7a_394a_4c1e_a29d_454df4a5766e.slice/crio-c864d6df491892ded3767273bdb3e8ecf2a4eed47ec40907aa79377790728781 WatchSource:0}: Error finding container c864d6df491892ded3767273bdb3e8ecf2a4eed47ec40907aa79377790728781: Status 404 returned error can't find the container with id c864d6df491892ded3767273bdb3e8ecf2a4eed47ec40907aa79377790728781 Oct 30 20:06:47.849828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:47.849801 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:06:48.434484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.434441 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:48.434657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.434524 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:48.504728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.503120 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" event={"ID":"f0ab3c7a-394a-4c1e-a29d-454df4a5766e","Type":"ContainerStarted","Data":"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1"} Oct 30 20:06:48.504728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.503199 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" event={"ID":"f0ab3c7a-394a-4c1e-a29d-454df4a5766e","Type":"ContainerStarted","Data":"c864d6df491892ded3767273bdb3e8ecf2a4eed47ec40907aa79377790728781"} Oct 30 20:06:48.504728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.504658 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:06:48.545154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:48.545081 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" podStartSLOduration=1.5450604220000002 podStartE2EDuration="1.545060422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:48.544355122 +0000 UTC m=+1875.993847872" watchObservedRunningTime="2026-10-30 20:06:48.545060422 +0000 UTC m=+1875.994553151" Oct 30 20:06:49.434435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:49.434380 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:06:49.434603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:49.434450 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:06:50.302823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:50.302793 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:06:50.303345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:50.302899 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:06:50.440170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:50.440136 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:50.440654 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:50.440638 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:50.441678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:50.441655 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:06:50.884959 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:06:50.884904 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:06:53.916997 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:53.916960 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:06:53.922557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:53.922532 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:53.941620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:53.941594 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7b5635-openshift-default-dockercfg-6r2j2\"" Oct 30 20:06:53.959663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:53.959625 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:06:54.007745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.007695 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.007947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.007815 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.007947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.007895 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.007950 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.007970 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008175 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.008155 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.008197 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm2l\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.008285 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.008407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.008321 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109242 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109291 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm2l\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109331 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109349 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109419 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109479 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109532 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109565 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109585 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109815 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109860 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.109953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.109925 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.110102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.110059 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.110229 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.110207 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.111960 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.111936 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.112076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.112027 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.130713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.130682 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.130872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.130771 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm2l\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.237074 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.237044 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:54.564029 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:06:54.563993 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0123ae97_5cff_49f0_ab2b_c8261ef9bf7a.slice/crio-cc1a26818fdc017a293cd2ed7b55addba649cb8116af186b6e6dbdec952fbe11 WatchSource:0}: Error finding container cc1a26818fdc017a293cd2ed7b55addba649cb8116af186b6e6dbdec952fbe11: Status 404 returned error can't find the container with id cc1a26818fdc017a293cd2ed7b55addba649cb8116af186b6e6dbdec952fbe11 Oct 30 20:06:54.565097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.565068 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:06:54.567538 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.567504 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:54.567617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.567573 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:54.567617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:54.567607 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:55.562895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:55.562849 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" event={"ID":"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a","Type":"ContainerStarted","Data":"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809"} Oct 30 20:06:55.563443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:55.562956 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" event={"ID":"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a","Type":"ContainerStarted","Data":"cc1a26818fdc017a293cd2ed7b55addba649cb8116af186b6e6dbdec952fbe11"} Oct 30 20:06:55.634567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:55.634507 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podStartSLOduration=2.634491343 podStartE2EDuration="2.634491343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:55.623120513 +0000 UTC m=+1883.072613239" watchObservedRunningTime="2026-10-30 20:06:55.634491343 +0000 UTC m=+1883.083984068" Oct 30 20:06:56.237697 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:56.237665 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:06:56.239742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:56.239712 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:06:56.239857 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:56.239773 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 20:06:57.237585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:57.237539 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:06:57.237981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:57.237599 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 20:06:58.237655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:58.237617 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:06:58.238170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:58.237678 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 20:06:59.238346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:59.238297 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:06:59.238795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:06:59.238383 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 20:07:00.238058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:00.238016 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:07:00.238220 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:00.238099 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 20:07:00.524272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:00.524187 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:07:01.386481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:01.386434 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:07:01.386980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:01.386959 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:07:01.388223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:01.388187 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:07:16.312888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.312814 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:07:16.319821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.319794 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.350461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.350427 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:07:16.355707 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.355676 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6cd2ad-openshift-default-dockercfg-r568l\"" Oct 30 20:07:16.423698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.423664 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.423867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.423778 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.423867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.423804 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.423867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.423864 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.424019 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.424006 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.424116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.424092 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.424180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.424137 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgj8\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.424288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.424274 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.424336 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.424302 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.525563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525525 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.525563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525567 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgj8\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.525831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525696 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.525831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525732 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.525831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525765 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525883 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525910 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525961 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.525963 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.526177 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.526202 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.526928 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.526890 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.527521 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.527493 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.527771 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.527741 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.529399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.529339 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.529594 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.529571 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.545483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.545446 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgj8\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.545649 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.545630 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.636200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.636097 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:16.971010 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:07:16.970980 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99e3810_2c66_4fd5_bbcf_c3785d8b7bda.slice/crio-3f921aac48d9e61b734feb2dd5f5c2a29d6c35ded8e3cb4428ce8489b442589c WatchSource:0}: Error finding container 3f921aac48d9e61b734feb2dd5f5c2a29d6c35ded8e3cb4428ce8489b442589c: Status 404 returned error can't find the container with id 3f921aac48d9e61b734feb2dd5f5c2a29d6c35ded8e3cb4428ce8489b442589c Oct 30 20:07:16.972237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.972212 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:07:16.974536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.974499 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:16.974643 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.974578 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:16.974643 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:16.974628 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:17.783319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:17.783264 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" event={"ID":"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda","Type":"ContainerStarted","Data":"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3"} Oct 30 20:07:17.783791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:17.783347 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" event={"ID":"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda","Type":"ContainerStarted","Data":"3f921aac48d9e61b734feb2dd5f5c2a29d6c35ded8e3cb4428ce8489b442589c"} Oct 30 20:07:17.876410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:17.866182 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podStartSLOduration=1.866160583 podStartE2EDuration="1.866160583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:17.851060835 +0000 UTC m=+1905.300553559" watchObservedRunningTime="2026-10-30 20:07:17.866160583 +0000 UTC m=+1905.315653309" Oct 30 20:07:18.637001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:18.636964 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:18.638891 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:18.638856 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.639029 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:18.638906 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:18.999992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:18.999961 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:07:19.005226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.005204 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.032239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.032205 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ebe57b-openshift-default-dockercfg-556nv\"" Oct 30 20:07:19.037403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.037373 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:07:19.096648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.096612 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.096849 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.096751 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.096849 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.096786 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.096964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.096850 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.097017 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.096975 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.097114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.097014 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.097114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.097045 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26w6p\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.097114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.097100 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.097219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.097176 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.219709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.219658 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.219920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.219725 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.219920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.219768 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26w6p\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.219920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.219830 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.219929 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220007 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220109 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220138 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220167 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220649 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.220925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.220898 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.221041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.221012 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.221328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.221304 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.222393 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.222351 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.222864 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.222845 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.224746 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.224723 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.244419 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.243597 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.245295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.245275 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26w6p\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.321702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.321627 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:19.637461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.637385 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:19.637461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.637438 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:19.693938 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:07:19.693909 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8271e7d_15c4_43ce_8b7a_04ed6db7afc0.slice/crio-bab573665d6765d93da83b911e9f637a528cb37c1cec715ae3975384c12b138f WatchSource:0}: Error finding container bab573665d6765d93da83b911e9f637a528cb37c1cec715ae3975384c12b138f: Status 404 returned error can't find the container with id bab573665d6765d93da83b911e9f637a528cb37c1cec715ae3975384c12b138f Oct 30 20:07:19.694994 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.694971 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:07:19.696550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.696517 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.696660 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.696591 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.696660 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.696637 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.811928 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.811888 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" event={"ID":"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0","Type":"ContainerStarted","Data":"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8"} Oct 30 20:07:19.812047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.811951 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" event={"ID":"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0","Type":"ContainerStarted","Data":"bab573665d6765d93da83b911e9f637a528cb37c1cec715ae3975384c12b138f"} Oct 30 20:07:19.861238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:19.861177 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podStartSLOduration=1.861158801 podStartE2EDuration="1.861158801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:19.857087282 +0000 UTC m=+1907.306580006" watchObservedRunningTime="2026-10-30 20:07:19.861158801 +0000 UTC m=+1907.310651529" Oct 30 20:07:20.322192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:20.322163 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:20.323671 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:20.323645 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.323781 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:20.323691 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:20.637930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:20.637827 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.638258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:20.638221 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:20.920423 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:07:20.920316 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:07:21.322914 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:21.322876 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:21.323419 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:21.322932 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:21.637401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:21.637292 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:21.637401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:21.637355 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:22.322173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:22.322134 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:22.322401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:22.322195 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:22.637768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:22.637687 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:22.637768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:22.637743 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:23.322812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:23.322766 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:23.322995 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:23.322834 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:23.636880 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:23.636796 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:07:23.636880 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:23.636857 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 20:07:24.322755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:24.322721 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:24.323172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:24.322776 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:24.644026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:24.643942 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:24.645182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:24.645144 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:24.646511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:24.646490 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:07:25.322098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:25.322065 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:25.322357 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:25.322331 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:26.327648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:26.327592 5991 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 20:07:26.329832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:26.328355 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 20:07:27.329055 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:27.329019 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:27.329574 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:27.329478 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:27.330181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:07:27.330164 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:07:50.870784 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:07:50.870743 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:08:01.460882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.460837 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:08:01.470249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.470183 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.501558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.501524 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:08:01.600698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.600655 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.600900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.600719 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.600900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.600853 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99c7\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.600900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.600898 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.601024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.601006 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.601069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.601054 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.601208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.601195 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.601248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.601224 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.601416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.601357 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.702815 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.702778 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99c7\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.702825 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.702902 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.702966 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703217 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703109 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703217 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703147 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703332 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703296 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703346 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703354 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703399 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703576 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703409 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703659 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703799 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.703970 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.703949 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.705885 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.705858 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.705980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.705962 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.727025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.726995 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.727201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.727090 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99c7\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:01.802494 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:01.802438 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:02.127317 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:08:02.127278 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc85d5fa_ec65_4a88_84a3_fa10887840fd.slice/crio-00905c7769233a04372dd55526bf581e982da5bb97049fed1163d2b60c9deceb WatchSource:0}: Error finding container 00905c7769233a04372dd55526bf581e982da5bb97049fed1163d2b60c9deceb: Status 404 returned error can't find the container with id 00905c7769233a04372dd55526bf581e982da5bb97049fed1163d2b60c9deceb Oct 30 20:08:02.128296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.128267 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:08:02.130413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.130348 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:02.130501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.130450 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:02.130501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.130480 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:02.255320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.255275 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" event={"ID":"dc85d5fa-ec65-4a88-84a3-fa10887840fd","Type":"ContainerStarted","Data":"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd"} Oct 30 20:08:02.255550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.255351 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" event={"ID":"dc85d5fa-ec65-4a88-84a3-fa10887840fd","Type":"ContainerStarted","Data":"00905c7769233a04372dd55526bf581e982da5bb97049fed1163d2b60c9deceb"} Oct 30 20:08:02.307055 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.306991 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podStartSLOduration=1.30697123 podStartE2EDuration="1.30697123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:02.301500341 +0000 UTC m=+1949.750993069" watchObservedRunningTime="2026-10-30 20:08:02.30697123 +0000 UTC m=+1949.756463955" Oct 30 20:08:02.802795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:02.802743 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:03.805926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:03.805881 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:03.806377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:03.805957 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:04.803635 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:04.803596 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:04.803822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:04.803656 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:05.803656 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:05.803621 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:05.804087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:05.803676 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:06.803804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:06.803764 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:06.804207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:06.803837 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:07.154600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:07.153615 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:08:07.154600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:07.153860 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" containerID="cri-o://20437a1787977e89748a8e6d27d3ca3a4795846437c03ced9dcca4202966a676" gracePeriod=30 Oct 30 20:08:07.803754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:07.803713 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:07.803956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:07.803790 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:08.803748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:08.803705 5991 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:08:08.803967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:08.803773 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 20:08:09.808339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:09.808303 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:09.808781 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:09.808652 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:09.809452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:09.809433 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:08:12.313411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.310028 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:08:12.436107 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.436028 5991 generic.go:356] "Generic (PLEG): container finished" podID="f1554151-0309-4745-b68e-7c37f5cde68e" containerID="20437a1787977e89748a8e6d27d3ca3a4795846437c03ced9dcca4202966a676" exitCode=0 Oct 30 20:08:12.436107 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.436065 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" event={"ID":"f1554151-0309-4745-b68e-7c37f5cde68e","Type":"ContainerDied","Data":"20437a1787977e89748a8e6d27d3ca3a4795846437c03ced9dcca4202966a676"} Oct 30 20:08:12.613428 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.613400 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:08:12.730887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.730853 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.730959 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.730997 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731151 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731186 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.731331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731193 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731249 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731473 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731478 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.731619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731558 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:12.731619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731569 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731600 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\") pod \"f1554151-0309-4745-b68e-7c37f5cde68e\" (UID: \"f1554151-0309-4745-b68e-7c37f5cde68e\") " Oct 30 20:08:12.731787 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731726 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.731981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.731957 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.733423 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.733348 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1554151-0309-4745-b68e-7c37f5cde68e-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.733873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.733856 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.733972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.733925 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.733972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.733943 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.733972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.733958 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.734392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.734342 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.735790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.735761 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:12.735932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.735904 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "kube-api-access-dgbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:12.736150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.736128 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo" pod "f1554151-0309-4745-b68e-7c37f5cde68e" (UID: "f1554151-0309-4745-b68e-7c37f5cde68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:12.835275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.835232 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbkd\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-kube-api-access-dgbkd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.835275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.835269 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1554151-0309-4745-b68e-7c37f5cde68e-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.835275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.835280 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1554151-0309-4745-b68e-7c37f5cde68e-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.835275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:12.835289 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1554151-0309-4745-b68e-7c37f5cde68e-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:13.446583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:13.446544 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" Oct 30 20:08:13.446583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:13.446558 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq" event={"ID":"f1554151-0309-4745-b68e-7c37f5cde68e","Type":"ContainerDied","Data":"fcbbe976413a1669f3c7028548b41d5c667c3c4bdff0998f66c93ddc1da3e10f"} Oct 30 20:08:13.447315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:13.446659 5991 scope.go:122] "RemoveContainer" containerID="20437a1787977e89748a8e6d27d3ca3a4795846437c03ced9dcca4202966a676" Oct 30 20:08:13.517236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:13.517206 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:08:13.529783 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:13.529750 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-w6wnq"] Oct 30 20:08:14.714468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:14.714418 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" path="/var/lib/kubelet/pods/f1554151-0309-4745-b68e-7c37f5cde68e/volumes" Oct 30 20:08:18.034997 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:18.034956 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 20:08:18.076943 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:18.076894 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 20:08:20.871971 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:08:20.871934 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:08:23.036176 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:23.036123 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" containerID="cri-o://9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0" gracePeriod=25 Oct 30 20:08:23.077567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:23.077518 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" containerID="cri-o://87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43" gracePeriod=25 Oct 30 20:08:24.833558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:24.833521 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:24.833558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:24.833549 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:34.843886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:34.843842 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:34.846269 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:34.846231 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:43.389295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.389259 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 20:08:43.438270 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.438236 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 20:08:43.512768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.512723 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\" (UniqueName: \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\") pod \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " Oct 30 20:08:43.512952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.512879 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\" (UniqueName: \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\") pod \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " Oct 30 20:08:43.513200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.513179 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\") pod \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\" (UID: \"88763cf9-b783-4581-b1ed-fcaeaeb26a75\") " Oct 30 20:08:43.513274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.513252 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\") pod \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\" (UID: \"c04d33bc-e2e6-4aa5-938e-8661fe22a136\") " Oct 30 20:08:43.513783 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.513604 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume" pod "88763cf9-b783-4581-b1ed-fcaeaeb26a75" (UID: "88763cf9-b783-4581-b1ed-fcaeaeb26a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:43.513783 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.513670 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume" pod "c04d33bc-e2e6-4aa5-938e-8661fe22a136" (UID: "c04d33bc-e2e6-4aa5-938e-8661fe22a136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:43.514206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.514182 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d33bc-e2e6-4aa5-938e-8661fe22a136-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:43.514285 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.514213 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88763cf9-b783-4581-b1ed-fcaeaeb26a75-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:43.515164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.515139 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2" pod "88763cf9-b783-4581-b1ed-fcaeaeb26a75" (UID: "88763cf9-b783-4581-b1ed-fcaeaeb26a75"). InnerVolumeSpecName "kube-api-access-hf7b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:43.515346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.515328 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd" pod "c04d33bc-e2e6-4aa5-938e-8661fe22a136" (UID: "c04d33bc-e2e6-4aa5-938e-8661fe22a136"). InnerVolumeSpecName "kube-api-access-n89xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:43.616033 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.615930 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89xd\" (UniqueName: \"kubernetes.io/projected/c04d33bc-e2e6-4aa5-938e-8661fe22a136-kube-api-access-n89xd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:43.616033 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.615967 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf7b2\" (UniqueName: \"kubernetes.io/projected/88763cf9-b783-4581-b1ed-fcaeaeb26a75-kube-api-access-hf7b2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:08:43.900866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.900773 5991 generic.go:356] "Generic (PLEG): container finished" podID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerID="87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43" exitCode=0 Oct 30 20:08:43.900866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.900823 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" Oct 30 20:08:43.900866 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.900854 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" event={"ID":"c04d33bc-e2e6-4aa5-938e-8661fe22a136","Type":"ContainerDied","Data":"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43"} Oct 30 20:08:43.901216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.900907 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml" event={"ID":"c04d33bc-e2e6-4aa5-938e-8661fe22a136","Type":"ContainerDied","Data":"cf68faed11dcbf97cba4fa4aac465e7d1d8305907eacf31a37d725d15d7061d1"} Oct 30 20:08:43.901216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.900932 5991 scope.go:122] "RemoveContainer" containerID="87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43" Oct 30 20:08:43.902747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.902721 5991 generic.go:356] "Generic (PLEG): container finished" podID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerID="9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0" exitCode=0 Oct 30 20:08:43.902887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.902775 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" Oct 30 20:08:43.902887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.902802 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" event={"ID":"88763cf9-b783-4581-b1ed-fcaeaeb26a75","Type":"ContainerDied","Data":"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0"} Oct 30 20:08:43.902887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.902859 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4" event={"ID":"88763cf9-b783-4581-b1ed-fcaeaeb26a75","Type":"ContainerDied","Data":"dc0648ca1c91ebd93c338227450a8b3ce7ae4a286a7d4b43abcc3be3d5d0e49c"} Oct 30 20:08:43.913300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.913281 5991 scope.go:122] "RemoveContainer" containerID="87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43" Oct 30 20:08:43.913695 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:08:43.913673 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43\": container with ID starting with 87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43 not found: ID does not exist" containerID="87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43" Oct 30 20:08:43.913759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.913704 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43"} err="failed to get container status \"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43\": rpc error: code = NotFound desc = could not find container \"87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43\": container with ID starting with 87ce778b6d6bd91b18d9d6416d9ecdd720a2e2c7a07ac41de08b73c2198aff43 not found: ID does not exist" Oct 30 20:08:43.913759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.913724 5991 scope.go:122] "RemoveContainer" containerID="9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0" Oct 30 20:08:43.925621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.925592 5991 scope.go:122] "RemoveContainer" containerID="9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0" Oct 30 20:08:43.925966 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:08:43.925948 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0\": container with ID starting with 9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0 not found: ID does not exist" containerID="9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0" Oct 30 20:08:43.926035 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.925977 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0"} err="failed to get container status \"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0\": rpc error: code = NotFound desc = could not find container \"9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0\": container with ID starting with 9c00d5e0a3a8fdfc02f07806e72695d5432a20830d0076f63ddd93b9ce7ce4d0 not found: ID does not exist" Oct 30 20:08:43.974503 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.974467 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 20:08:43.981537 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:43.981504 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb16ce-bcd9dbdc9-vmwml"] Oct 30 20:08:44.022714 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:44.022679 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 20:08:44.026708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:44.026679 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb16ce-66cb7d76c8-z54x4"] Oct 30 20:08:44.681922 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:44.681883 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" path="/var/lib/kubelet/pods/88763cf9-b783-4581-b1ed-fcaeaeb26a75/volumes" Oct 30 20:08:44.682593 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:08:44.682573 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" path="/var/lib/kubelet/pods/c04d33bc-e2e6-4aa5-938e-8661fe22a136/volumes" Oct 30 20:08:50.871196 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:08:50.871155 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:09:01.012704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:01.012669 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 20:09:01.013258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:01.012889 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" containerID="cri-o://2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf" gracePeriod=30 Oct 30 20:09:01.068588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:01.067654 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 20:09:01.069308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:01.069237 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" containerID="cri-o://1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617" gracePeriod=30 Oct 30 20:09:05.823669 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:05.823627 5991 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:09:05.824076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:05.823686 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:09:06.417817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.417784 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 20:09:06.484726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.484698 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 20:09:06.524406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524331 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524469 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524525 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524693 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524618 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524683 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.524743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524732 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524766 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524822 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.524934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.524890 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.525094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.525076 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\") pod \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\" (UID: \"9f0949fc-09a7-450b-94c8-6c4df5c323a7\") " Oct 30 20:09:06.525303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.525274 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.525654 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.525631 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.525993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.525967 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:06.526090 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.526045 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.526090 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.526066 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.526216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.526104 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.526685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.526659 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.527250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.527222 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.527400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.527329 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:06.528087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.528065 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "kube-api-access-wc4bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.528477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.528455 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token" pod "9f0949fc-09a7-450b-94c8-6c4df5c323a7" (UID: "9f0949fc-09a7-450b-94c8-6c4df5c323a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.626816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626779 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.626816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626821 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626861 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626895 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626931 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.626953 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627003 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627032 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627360 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627201 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.627448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627413 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.627538 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627498 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.627644 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627621 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\") pod \"6dbea309-e025-4968-b02c-f0b5c6913317\" (UID: \"6dbea309-e025-4968-b02c-f0b5c6913317\") " Oct 30 20:09:06.627728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627657 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:06.627865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.627840 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.628572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628525 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbea309-e025-4968-b02c-f0b5c6913317-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628555 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628569 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628582 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628595 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628611 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4bs\" (UniqueName: \"kubernetes.io/projected/9f0949fc-09a7-450b-94c8-6c4df5c323a7-kube-api-access-wc4bs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628625 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628638 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628652 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628665 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.628706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.628680 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0949fc-09a7-450b-94c8-6c4df5c323a7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.629167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.629145 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:06.629578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.629553 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.629578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.629574 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.629716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.629585 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm" pod "6dbea309-e025-4968-b02c-f0b5c6913317" (UID: "6dbea309-e025-4968-b02c-f0b5c6913317"). InnerVolumeSpecName "kube-api-access-zxbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.731049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.731013 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbea309-e025-4968-b02c-f0b5c6913317-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.731049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.731045 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbxm\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-kube-api-access-zxbxm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.731049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.731057 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbea309-e025-4968-b02c-f0b5c6913317-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.731295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:06.731066 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbea309-e025-4968-b02c-f0b5c6913317-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:07.125735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.125696 5991 generic.go:356] "Generic (PLEG): container finished" podID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerID="2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf" exitCode=0 Oct 30 20:09:07.126303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.125742 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" Oct 30 20:09:07.126303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.125764 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" event={"ID":"9f0949fc-09a7-450b-94c8-6c4df5c323a7","Type":"ContainerDied","Data":"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf"} Oct 30 20:09:07.126303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.125819 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj" event={"ID":"9f0949fc-09a7-450b-94c8-6c4df5c323a7","Type":"ContainerDied","Data":"b6fa4aa9b59de5aed095cb23cba97c5306dc4d2651c55e8a9bf0fafb189adf8c"} Oct 30 20:09:07.126303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.125844 5991 scope.go:122] "RemoveContainer" containerID="2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf" Oct 30 20:09:07.127681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.127653 5991 generic.go:356] "Generic (PLEG): container finished" podID="6dbea309-e025-4968-b02c-f0b5c6913317" containerID="1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617" exitCode=0 Oct 30 20:09:07.127813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.127699 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" Oct 30 20:09:07.127813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.127803 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" event={"ID":"6dbea309-e025-4968-b02c-f0b5c6913317","Type":"ContainerDied","Data":"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617"} Oct 30 20:09:07.127945 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.127852 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786" event={"ID":"6dbea309-e025-4968-b02c-f0b5c6913317","Type":"ContainerDied","Data":"95479ef14e35f36bb89b799197488715fe56c8c7f998c87b1a53a40ef3de07c4"} Oct 30 20:09:07.138747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.138719 5991 scope.go:122] "RemoveContainer" containerID="2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf" Oct 30 20:09:07.139094 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:07.139072 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf\": container with ID starting with 2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf not found: ID does not exist" containerID="2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf" Oct 30 20:09:07.139180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.139107 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf"} err="failed to get container status \"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf\": rpc error: code = NotFound desc = could not find container \"2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf\": container with ID starting with 2f439b1495ee8e5091f6e25ac9d62d0192d5634b25d0800953e4d5cf4acd97cf not found: ID does not exist" Oct 30 20:09:07.139180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.139134 5991 scope.go:122] "RemoveContainer" containerID="1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617" Oct 30 20:09:07.152887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.152865 5991 scope.go:122] "RemoveContainer" containerID="1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617" Oct 30 20:09:07.153312 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:07.153279 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617\": container with ID starting with 1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617 not found: ID does not exist" containerID="1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617" Oct 30 20:09:07.153474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.153321 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617"} err="failed to get container status \"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617\": rpc error: code = NotFound desc = could not find container \"1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617\": container with ID starting with 1f7e916209260d44cc0ab8fe35b31c670834e5dfefc7aa558e7262443eaa9617 not found: ID does not exist" Oct 30 20:09:07.197407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.194888 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 20:09:07.209396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.209346 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-8h786"] Oct 30 20:09:07.240918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.240884 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 20:09:07.247773 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:07.247734 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-lvflj"] Oct 30 20:09:08.680793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:08.680757 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" path="/var/lib/kubelet/pods/6dbea309-e025-4968-b02c-f0b5c6913317/volumes" Oct 30 20:09:08.681288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:08.681272 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" path="/var/lib/kubelet/pods/9f0949fc-09a7-450b-94c8-6c4df5c323a7/volumes" Oct 30 20:09:08.998536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:08.998323 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 20:09:09.017628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.017580 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" podUID="a90bf0d9-749f-4791-b7b9-235f10512642" containerName="maas-api" containerID="cri-o://67cd7072da1853404c5bc458a49691bebc1ea9baba7f6cdd1cd6a675527f884f" gracePeriod=30 Oct 30 20:09:09.150178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.150144 5991 generic.go:356] "Generic (PLEG): container finished" podID="a90bf0d9-749f-4791-b7b9-235f10512642" containerID="67cd7072da1853404c5bc458a49691bebc1ea9baba7f6cdd1cd6a675527f884f" exitCode=0 Oct 30 20:09:09.150406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.150219 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" event={"ID":"a90bf0d9-749f-4791-b7b9-235f10512642","Type":"ContainerDied","Data":"67cd7072da1853404c5bc458a49691bebc1ea9baba7f6cdd1cd6a675527f884f"} Oct 30 20:09:09.397152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.397121 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 20:09:09.581303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.581262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\" (UniqueName: \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\") pod \"a90bf0d9-749f-4791-b7b9-235f10512642\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " Oct 30 20:09:09.581555 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.581435 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\") pod \"a90bf0d9-749f-4791-b7b9-235f10512642\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " Oct 30 20:09:09.581555 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.581507 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\") pod \"a90bf0d9-749f-4791-b7b9-235f10512642\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " Oct 30 20:09:09.636965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.630526 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") pod \"a90bf0d9-749f-4791-b7b9-235f10512642\" (UID: \"a90bf0d9-749f-4791-b7b9-235f10512642\") " Oct 30 20:09:09.636965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.582539 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca" pod "a90bf0d9-749f-4791-b7b9-235f10512642" (UID: "a90bf0d9-749f-4791-b7b9-235f10512642"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:09.636965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.634968 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm" pod "a90bf0d9-749f-4791-b7b9-235f10512642" (UID: "a90bf0d9-749f-4791-b7b9-235f10512642"). InnerVolumeSpecName "kube-api-access-r9jxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:09.636965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.635033 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs" pod "a90bf0d9-749f-4791-b7b9-235f10512642" (UID: "a90bf0d9-749f-4791-b7b9-235f10512642"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:09.664390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.660617 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls" pod "a90bf0d9-749f-4791-b7b9-235f10512642" (UID: "a90bf0d9-749f-4791-b7b9-235f10512642"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:09.736390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.733667 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jxm\" (UniqueName: \"kubernetes.io/projected/a90bf0d9-749f-4791-b7b9-235f10512642-kube-api-access-r9jxm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.736390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.733706 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90bf0d9-749f-4791-b7b9-235f10512642-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.736390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.733723 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.736390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:09.733736 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90bf0d9-749f-4791-b7b9-235f10512642-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:10.160630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.160597 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" Oct 30 20:09:10.160804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.160599 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm" event={"ID":"a90bf0d9-749f-4791-b7b9-235f10512642","Type":"ContainerDied","Data":"eae491cb04600b11ffc171dc0f6b26913d3b288445eebe789edc9c23cad980cf"} Oct 30 20:09:10.160804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.160738 5991 scope.go:122] "RemoveContainer" containerID="67cd7072da1853404c5bc458a49691bebc1ea9baba7f6cdd1cd6a675527f884f" Oct 30 20:09:10.220349 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.220315 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 20:09:10.232073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.232041 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb16ce-d8bfb88dc-85plm"] Oct 30 20:09:10.681469 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:10.681433 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90bf0d9-749f-4791-b7b9-235f10512642" path="/var/lib/kubelet/pods/a90bf0d9-749f-4791-b7b9-235f10512642/volumes" Oct 30 20:09:18.722482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:18.722446 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:09:18.723430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:18.723392 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" containerID="cri-o://9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190" gracePeriod=30 Oct 30 20:09:20.444007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:20.443967 5991 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:09:20.444409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:20.444032 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:09:20.871488 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:20.871447 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:09:24.017355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.017326 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:09:24.198606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198569 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.198814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198647 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.198814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198782 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.198927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198847 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.198927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198883 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.198927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198878 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:24.199097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198941 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:24.199097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198957 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.199097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.198988 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.199720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.199098 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.199720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.199182 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:24.199720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.199301 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\") pod \"3af73142-cdcb-45b6-997f-b5f23d759b7d\" (UID: \"3af73142-cdcb-45b6-997f-b5f23d759b7d\") " Oct 30 20:09:24.199720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.199493 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:24.199924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.199777 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:24.200186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.200163 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af73142-cdcb-45b6-997f-b5f23d759b7d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.200261 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.200192 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.200261 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.200207 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.200261 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.200220 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.200261 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.200232 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.201802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.201775 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:24.202214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.202188 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "kube-api-access-4h5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:24.202286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.202240 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:24.203723 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.203691 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy" pod "3af73142-cdcb-45b6-997f-b5f23d759b7d" (UID: "3af73142-cdcb-45b6-997f-b5f23d759b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:24.281750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.281670 5991 generic.go:356] "Generic (PLEG): container finished" podID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerID="9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190" exitCode=0 Oct 30 20:09:24.281750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.281723 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" Oct 30 20:09:24.281949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.281757 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" event={"ID":"3af73142-cdcb-45b6-997f-b5f23d759b7d","Type":"ContainerDied","Data":"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190"} Oct 30 20:09:24.281949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.281814 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5" event={"ID":"3af73142-cdcb-45b6-997f-b5f23d759b7d","Type":"ContainerDied","Data":"74e417c8a3b9f905b1e98f1149014f22d536927099eb48420ce9bf6fa1413a99"} Oct 30 20:09:24.281949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.281836 5991 scope.go:122] "RemoveContainer" containerID="9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190" Oct 30 20:09:24.294556 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.294539 5991 scope.go:122] "RemoveContainer" containerID="9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190" Oct 30 20:09:24.294828 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:24.294807 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190\": container with ID starting with 9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190 not found: ID does not exist" containerID="9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190" Oct 30 20:09:24.294885 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.294836 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190"} err="failed to get container status \"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190\": rpc error: code = NotFound desc = could not find container \"9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190\": container with ID starting with 9261f6dd25e1c9ab5b2392cf1233d7ddba9c95ffec6822946d3d3e92578f9190 not found: ID does not exist" Oct 30 20:09:24.301930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.301907 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.301930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.301931 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.302056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.301939 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af73142-cdcb-45b6-997f-b5f23d759b7d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.302056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.301948 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5mt\" (UniqueName: \"kubernetes.io/projected/3af73142-cdcb-45b6-997f-b5f23d759b7d-kube-api-access-4h5mt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:24.336683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.336590 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:09:24.340584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.340558 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-mwkl5"] Oct 30 20:09:24.680539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:24.680456 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" path="/var/lib/kubelet/pods/3af73142-cdcb-45b6-997f-b5f23d759b7d/volumes" Oct 30 20:09:34.967596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:34.967560 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:09:35.134803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.134763 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:09:35.135645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135629 5991 cpu_manager.go:414] "containerMap: removing container" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135646 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135666 5991 cpu_manager.go:414] "containerMap: removing container" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135672 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135681 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135686 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" Oct 30 20:09:35.135718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135718 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a90bf0d9-749f-4791-b7b9-235f10512642" containerName="maas-api" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135723 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90bf0d9-749f-4791-b7b9-235f10512642" containerName="maas-api" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135734 5991 cpu_manager.go:414] "containerMap: removing container" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135739 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135749 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135753 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135766 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135771 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135880 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0949fc-09a7-450b-94c8-6c4df5c323a7" containerName="istio-proxy" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135891 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbea309-e025-4968-b02c-f0b5c6913317" containerName="istio-proxy" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135898 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af73142-cdcb-45b6-997f-b5f23d759b7d" containerName="istio-proxy" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135909 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90bf0d9-749f-4791-b7b9-235f10512642" containerName="maas-api" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135916 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="88763cf9-b783-4581-b1ed-fcaeaeb26a75" containerName="payload-processing" Oct 30 20:09:35.135919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135921 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1554151-0309-4745-b68e-7c37f5cde68e" containerName="istio-proxy" Oct 30 20:09:35.136299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.135929 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04d33bc-e2e6-4aa5-938e-8661fe22a136" containerName="payload-pre-processing" Oct 30 20:09:35.141471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.141448 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.159602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.159571 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4cfcee-serving-cert\"" Oct 30 20:09:35.160852 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.160827 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:09:35.289748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.289708 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.289967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.289922 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.290058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.290037 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.290114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.290099 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbjk\" (UniqueName: \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.391731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.391690 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.391936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.391805 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.391936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.391857 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbjk\" (UniqueName: \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.392094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.392077 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.393027 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.392997 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.394732 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.394695 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.394847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.394755 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.411560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.411531 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbjk\" (UniqueName: \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\") pod \"maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.457128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.457088 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:35.775342 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:09:35.775315 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2eff3d8_9984_478f_a4e8_98b23b885c2a.slice/crio-2d6023f322b2b355bf871e636ad92e4670c9fbb8ef801f59b0e5be0d062a2da5 WatchSource:0}: Error finding container 2d6023f322b2b355bf871e636ad92e4670c9fbb8ef801f59b0e5be0d062a2da5: Status 404 returned error can't find the container with id 2d6023f322b2b355bf871e636ad92e4670c9fbb8ef801f59b0e5be0d062a2da5 Oct 30 20:09:35.776343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:35.776320 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:09:36.392718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:36.392681 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" event={"ID":"a2eff3d8-9984-478f-a4e8-98b23b885c2a","Type":"ContainerStarted","Data":"2d6023f322b2b355bf871e636ad92e4670c9fbb8ef801f59b0e5be0d062a2da5"} Oct 30 20:09:36.610919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:36.610837 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 20:09:36.611068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:36.611024 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerName="discovery" containerID="cri-o://fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734" gracePeriod=30 Oct 30 20:09:36.904968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:36.904931 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 20:09:37.029049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029006 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029230 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029206 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029398 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029438 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029521 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029564 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029633 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029622 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\") pod \"dd83d175-5490-4ebd-a249-60d1c7851aba\" (UID: \"dd83d175-5490-4ebd-a249-60d1c7851aba\") " Oct 30 20:09:37.029684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.029643 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:37.030409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.030383 5991 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-ca-configmap\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.031882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.031844 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "kube-api-access-4dqw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:37.032191 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.032165 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:37.032272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.032193 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:37.032346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.032304 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:37.032475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.032455 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:37.032650 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.032624 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs" pod "dd83d175-5490-4ebd-a249-60d1c7851aba" (UID: "dd83d175-5490-4ebd-a249-60d1c7851aba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132591 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqw7\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-kube-api-access-4dqw7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132619 5991 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-csr-dns-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132628 5991 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-cacerts\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132637 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132646 5991 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd83d175-5490-4ebd-a249-60d1c7851aba-local-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.132667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.132653 5991 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd83d175-5490-4ebd-a249-60d1c7851aba-istio-kubeconfig\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.403122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.403022 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" event={"ID":"a2eff3d8-9984-478f-a4e8-98b23b885c2a","Type":"ContainerStarted","Data":"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6"} Oct 30 20:09:37.403122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.403099 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:37.404766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.404735 5991 generic.go:356] "Generic (PLEG): container finished" podID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerID="fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734" exitCode=0 Oct 30 20:09:37.404906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.404767 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" Oct 30 20:09:37.404906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.404767 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" event={"ID":"dd83d175-5490-4ebd-a249-60d1c7851aba","Type":"ContainerDied","Data":"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734"} Oct 30 20:09:37.404906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.404824 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d" event={"ID":"dd83d175-5490-4ebd-a249-60d1c7851aba","Type":"ContainerDied","Data":"17e9dc18771a5943dcbe596f5a555e9462b402aa4fbe3590a00c2ab76473ddcd"} Oct 30 20:09:37.404906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.404850 5991 scope.go:122] "RemoveContainer" containerID="fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734" Oct 30 20:09:37.418018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.417997 5991 scope.go:122] "RemoveContainer" containerID="fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734" Oct 30 20:09:37.418397 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:37.418350 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734\": container with ID starting with fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734 not found: ID does not exist" containerID="fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734" Oct 30 20:09:37.418508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.418405 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734"} err="failed to get container status \"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734\": rpc error: code = NotFound desc = could not find container \"fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734\": container with ID starting with fb0490e49570380436b7dab376289ce5de3767cf2dca9659e079e711b34dc734 not found: ID does not exist" Oct 30 20:09:37.466998 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.466942 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" podStartSLOduration=1.922056499 podStartE2EDuration="2.466927423s" totalImagesPullingTime="544.870924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:37.464978835 +0000 UTC m=+2044.914471570" watchObservedRunningTime="2026-10-30 20:09:37.466927423 +0000 UTC m=+2044.916420148" Oct 30 20:09:37.511117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.511079 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 20:09:37.519394 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:37.519319 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q97d"] Oct 30 20:09:38.680907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:38.680870 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" path="/var/lib/kubelet/pods/dd83d175-5490-4ebd-a249-60d1c7851aba/volumes" Oct 30 20:09:43.420827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:43.420790 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:09:50.871344 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:09:50.871296 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:09:57.976829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:57.976796 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:09:57.977661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:57.977644 5991 cpu_manager.go:414] "containerMap: removing container" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerName="discovery" Oct 30 20:09:57.977661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:57.977662 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerName="discovery" Oct 30 20:09:57.977819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:57.977809 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd83d175-5490-4ebd-a249-60d1c7851aba" containerName="discovery" Oct 30 20:09:57.981841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:57.981814 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.005685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.005642 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:09:58.012064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.012027 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-21c4ab\"/\"openshift-service-ca.crt\"" Oct 30 20:09:58.012274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.012105 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-21c4ab\"/\"kube-root-ca.crt\"" Oct 30 20:09:58.012713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.012689 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-21c4ab\"/\"auth-test-model-21c4ab-kserve-self-signed-certs\"" Oct 30 20:09:58.013711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.013687 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-21c4ab\"/\"default-dockercfg-lbw6h\"" Oct 30 20:09:58.068407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068320 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.068607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068517 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.068607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068574 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.068694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068647 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.068847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068831 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dm5\" (UniqueName: \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.068918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.068863 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171002 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.170964 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dm5\" (UniqueName: \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171232 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171017 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171232 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171115 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171353 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171266 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171353 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171341 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171394 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171610 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171797 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171768 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.171797 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.171785 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.173986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.173957 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.174334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.174311 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.190515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.190483 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dm5\" (UniqueName: \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\") pod \"auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.300894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.300803 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:09:58.643062 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:58.643025 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:09:58.644227 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:09:58.644175 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b0b0519_3d28_4ec9_99c2_e2cebc3963c2.slice/crio-e564fe4f35c31d722a61e82e60cb5a7656687b0a193cbb86ea02dc487949cf67 WatchSource:0}: Error finding container e564fe4f35c31d722a61e82e60cb5a7656687b0a193cbb86ea02dc487949cf67: Status 404 returned error can't find the container with id e564fe4f35c31d722a61e82e60cb5a7656687b0a193cbb86ea02dc487949cf67 Oct 30 20:09:59.601797 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:59.601762 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" event={"ID":"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2","Type":"ContainerStarted","Data":"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3"} Oct 30 20:09:59.601797 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:59.601803 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" event={"ID":"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2","Type":"ContainerStarted","Data":"e564fe4f35c31d722a61e82e60cb5a7656687b0a193cbb86ea02dc487949cf67"} Oct 30 20:09:59.644193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:09:59.644135 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" podStartSLOduration=2.6441201149999998 podStartE2EDuration="2.644120115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:59.641754275 +0000 UTC m=+2067.091247266" watchObservedRunningTime="2026-10-30 20:09:59.644120115 +0000 UTC m=+2067.093612919" Oct 30 20:10:00.300886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:00.300854 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:10:02.046014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.045970 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:10:02.051148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.051118 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.070560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.070526 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d3c508-openshift-default-dockercfg-qmnlc\"" Oct 30 20:10:02.087331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.087296 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:10:02.133966 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.133924 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134051 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134113 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134182 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134244 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134280 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vd4\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134307 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134458 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.134570 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.134555 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.235837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.235797 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.235901 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236007 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236107 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236158 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236216 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236259 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236311 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236381 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vd4\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236424 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236661 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236773 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.236823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.236785 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.237109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.237089 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.238351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.238328 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.238629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.238611 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.255283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.255245 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.255474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.255389 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vd4\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.367518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.367434 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:02.699108 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:10:02.699073 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10d707c_0645_43dc_a977_84d82fd1478c.slice/crio-5f1a61e36236c762d8a08b5e3f9bb8a39231dc8b7ef49d0fc189f13b3fed1093 WatchSource:0}: Error finding container 5f1a61e36236c762d8a08b5e3f9bb8a39231dc8b7ef49d0fc189f13b3fed1093: Status 404 returned error can't find the container with id 5f1a61e36236c762d8a08b5e3f9bb8a39231dc8b7ef49d0fc189f13b3fed1093 Oct 30 20:10:02.700126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.700085 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:10:02.702872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.702831 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:02.703013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.702922 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:02.703013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:02.702971 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:03.636237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:03.636198 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" event={"ID":"c10d707c-0645-43dc-a977-84d82fd1478c","Type":"ContainerStarted","Data":"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38"} Oct 30 20:10:03.636756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:03.636251 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" event={"ID":"c10d707c-0645-43dc-a977-84d82fd1478c","Type":"ContainerStarted","Data":"5f1a61e36236c762d8a08b5e3f9bb8a39231dc8b7ef49d0fc189f13b3fed1093"} Oct 30 20:10:03.728061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:03.727998 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podStartSLOduration=1.727981928 podStartE2EDuration="1.727981928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:03.724930141 +0000 UTC m=+2071.174422865" watchObservedRunningTime="2026-10-30 20:10:03.727981928 +0000 UTC m=+2071.177474652" Oct 30 20:10:04.377400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:04.372803 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:04.390038 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:04.390000 5991 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:10:04.390228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:04.390067 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 20:10:05.383399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:05.374837 5991 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:10:05.383399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:05.374906 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 20:10:06.380391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:06.372851 5991 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:10:06.380391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:06.372921 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 20:10:07.380869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:07.380826 5991 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:10:07.382165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:07.382132 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 20:10:08.301940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:08.301897 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:10:08.317295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:08.317268 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:10:08.385389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:08.377846 5991 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:10:08.385389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:08.377913 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 20:10:08.711777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:08.711686 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:10:09.374101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:09.374067 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:09.374529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:09.374515 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:09.375497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:09.375472 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:10:20.870787 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:10:20.870700 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:10:33.079574 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.079539 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:10:33.084912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.084887 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:10:33.459825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.459735 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:10:33.465171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.465141 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.486773 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.486739 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:10:33.494572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.494543 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a64be2\"/\"openshift-service-ca.crt\"" Oct 30 20:10:33.494755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.494738 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a64be2\"/\"kube-root-ca.crt\"" Oct 30 20:10:33.494808 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.494793 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a64be2\"/\"default-dockercfg-zgl8n\"" Oct 30 20:10:33.494854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.494835 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a64be2\"/\"test-model-a64be2-kserve-self-signed-certs\"" Oct 30 20:10:33.561510 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561472 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.561708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561607 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.561708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561670 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwgq\" (UniqueName: \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.561791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561720 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.561791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561737 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.561862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.561831 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663107 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663199 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663240 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwgq\" (UniqueName: \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663288 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663304 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663344 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663701 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663676 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663704 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.663830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.663808 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.666052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.666029 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.666311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.666293 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.684704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.684663 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwgq\" (UniqueName: \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\") pod \"test-model-a64be2-kserve-5f975c5f5-wnn8w\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:33.785388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:33.782990 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:34.103676 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:10:34.103647 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84bb9568_16fc_4dc7_bf37_71520be39eb8.slice/crio-4ea7b5cd52c55b6f6882b2f5b410373e4a3409def9997837865fb1d006223b95 WatchSource:0}: Error finding container 4ea7b5cd52c55b6f6882b2f5b410373e4a3409def9997837865fb1d006223b95: Status 404 returned error can't find the container with id 4ea7b5cd52c55b6f6882b2f5b410373e4a3409def9997837865fb1d006223b95 Oct 30 20:10:34.104315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:34.104292 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:10:34.915132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:34.915097 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" event={"ID":"84bb9568-16fc-4dc7-bf37-71520be39eb8","Type":"ContainerStarted","Data":"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527"} Oct 30 20:10:34.915132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:34.915139 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" event={"ID":"84bb9568-16fc-4dc7-bf37-71520be39eb8","Type":"ContainerStarted","Data":"4ea7b5cd52c55b6f6882b2f5b410373e4a3409def9997837865fb1d006223b95"} Oct 30 20:10:35.786757 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.786711 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:35.913302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.913235 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" podStartSLOduration=2.913210213 podStartE2EDuration="2.913210213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:34.971871602 +0000 UTC m=+2102.421364331" watchObservedRunningTime="2026-10-30 20:10:35.913210213 +0000 UTC m=+2103.362702943" Oct 30 20:10:35.916208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.916180 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:10:35.921344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.921315 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:35.936821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.936790 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4cfcee\"/\"openshift-service-ca.crt\"" Oct 30 20:10:35.946848 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.945461 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:10:35.949140 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:35.949107 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4cfcee\"/\"kube-root-ca.crt\"" Oct 30 20:10:36.001309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001270 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.001510 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001353 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nl9z\" (UniqueName: \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.001639 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001612 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.001792 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001772 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.001913 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001896 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.002012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.001999 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.020299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.020249 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4cfcee\"/\"default-dockercfg-k6479\"" Oct 30 20:10:36.020512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.020316 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4cfcee\"/\"test-model-4cfcee-kserve-self-signed-certs\"" Oct 30 20:10:36.103424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103313 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103420 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nl9z\" (UniqueName: \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103486 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103611 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103710 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103781 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103803 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.103948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.103919 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.104077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.104056 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.105853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.105829 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.106163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.106146 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.123237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.123205 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nl9z\" (UniqueName: \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\") pod \"test-model-4cfcee-kserve-74fd4f447f-bg64x\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.246917 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.246882 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:36.561140 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:10:36.561105 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b2ff98_28ad_430a_b5cc_85281ffe12dc.slice/crio-b9f0878396f077ee0f8e42af89b0458a9978a02a761c7f5e1bc3c49c0635126d WatchSource:0}: Error finding container b9f0878396f077ee0f8e42af89b0458a9978a02a761c7f5e1bc3c49c0635126d: Status 404 returned error can't find the container with id b9f0878396f077ee0f8e42af89b0458a9978a02a761c7f5e1bc3c49c0635126d Oct 30 20:10:36.562875 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.562838 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:10:36.939116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.939006 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" event={"ID":"17b2ff98-28ad-430a-b5cc-85281ffe12dc","Type":"ContainerStarted","Data":"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9"} Oct 30 20:10:36.939116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.939080 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" event={"ID":"17b2ff98-28ad-430a-b5cc-85281ffe12dc","Type":"ContainerStarted","Data":"b9f0878396f077ee0f8e42af89b0458a9978a02a761c7f5e1bc3c49c0635126d"} Oct 30 20:10:36.983016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:36.982949 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" podStartSLOduration=1.982931417 podStartE2EDuration="1.982931417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:36.980070747 +0000 UTC m=+2104.429563475" watchObservedRunningTime="2026-10-30 20:10:36.982931417 +0000 UTC m=+2104.432424141" Oct 30 20:10:37.247161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:37.247119 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:43.786479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:43.786444 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:43.800774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:43.800741 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:44.027117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:44.027081 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:10:46.247839 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:46.247802 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:46.260869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:46.260837 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:47.063836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:47.063803 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:10:50.871409 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:10:50.871348 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:10:59.230960 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.230925 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:10:59.238124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.238091 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.253222 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.253192 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-936f70-openshift-default-dockercfg-jtq7n\"" Oct 30 20:10:59.271768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.271722 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:10:59.417847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.417795 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25sp\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.417847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.417855 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418081 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.417986 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418081 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418039 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418160 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418112 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418219 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418314 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418432 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.418538 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.418525 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.519966 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520064 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520160 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520206 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25sp\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520238 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520310 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520385 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520547 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520558 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520765 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520629 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520765 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520691 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520765 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520712 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.520993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.520966 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.521294 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.521273 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.522930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.522903 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.523116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.523094 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.566417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.566333 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25sp\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.579333 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.579297 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:10:59.855234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:10:59.855135 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:11:00.181009 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:11:00.180976 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfd2117_344b_4f12_95d3_242553727ad3.slice/crio-cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5 WatchSource:0}: Error finding container cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5: Status 404 returned error can't find the container with id cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5 Oct 30 20:11:00.181220 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:00.181199 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:11:00.184884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:00.184852 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:00.185004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:00.184923 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:00.185004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:00.184966 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:01.173988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.173945 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" event={"ID":"dcfd2117-344b-4f12-95d3-242553727ad3","Type":"ContainerStarted","Data":"bb496389bc3c225ca24c9d75bca883d90f7ca399d11a4cf1640cbbe3d4831d86"} Oct 30 20:11:01.174389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.174016 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" event={"ID":"dcfd2117-344b-4f12-95d3-242553727ad3","Type":"ContainerStarted","Data":"cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5"} Oct 30 20:11:01.258829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.258765 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podStartSLOduration=2.258747575 podStartE2EDuration="2.258747575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:01.238668854 +0000 UTC m=+2128.688161592" watchObservedRunningTime="2026-10-30 20:11:01.258747575 +0000 UTC m=+2128.708240300" Oct 30 20:11:01.856153 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.856113 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:11:01.858435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.858409 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:01.858565 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:01.858455 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:02.855921 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:02.855875 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:02.856406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:02.855963 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:03.856509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:03.856466 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:03.856961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:03.856554 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:04.856188 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:04.856148 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:04.856477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:04.856207 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:05.855853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:05.855811 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:05.856307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:05.855895 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:06.856325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:06.856290 5991 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 20:11:06.856878 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:06.856395 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 20:11:07.861243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:07.861213 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:11:07.861727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:07.861709 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:11:07.862411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:07.862388 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:11:18.723734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:11:18.723693 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:11:20.871703 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:11:20.871665 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:11:50.880219 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:11:50.880130 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:12:02.025569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:02.025524 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:12:02.057950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:02.057914 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:12:07.026592 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.026534 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" podUID="a50d6d58-4921-4961-b36c-a2158faa1b26" containerName="payload-processing" containerID="cri-o://231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527" gracePeriod=25 Oct 30 20:12:07.059146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.059031 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" containerID="cri-o://582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e" gracePeriod=25 Oct 30 20:12:07.345668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.345635 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:12:07.483685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.483647 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\" (UniqueName: \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\") pod \"a50d6d58-4921-4961-b36c-a2158faa1b26\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " Oct 30 20:12:07.483894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.483776 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\") pod \"a50d6d58-4921-4961-b36c-a2158faa1b26\" (UID: \"a50d6d58-4921-4961-b36c-a2158faa1b26\") " Oct 30 20:12:07.484258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.484225 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume" pod "a50d6d58-4921-4961-b36c-a2158faa1b26" (UID: "a50d6d58-4921-4961-b36c-a2158faa1b26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:07.484592 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.484568 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d6d58-4921-4961-b36c-a2158faa1b26-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:12:07.486029 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.486005 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c" pod "a50d6d58-4921-4961-b36c-a2158faa1b26" (UID: "a50d6d58-4921-4961-b36c-a2158faa1b26"). InnerVolumeSpecName "kube-api-access-2h87c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:07.586322 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.586231 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h87c\" (UniqueName: \"kubernetes.io/projected/a50d6d58-4921-4961-b36c-a2158faa1b26-kube-api-access-2h87c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:12:07.828242 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.828206 5991 generic.go:356] "Generic (PLEG): container finished" podID="a50d6d58-4921-4961-b36c-a2158faa1b26" containerID="231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527" exitCode=0 Oct 30 20:12:07.828472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.828257 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" Oct 30 20:12:07.828472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.828297 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" event={"ID":"a50d6d58-4921-4961-b36c-a2158faa1b26","Type":"ContainerDied","Data":"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527"} Oct 30 20:12:07.828472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.828357 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5" event={"ID":"a50d6d58-4921-4961-b36c-a2158faa1b26","Type":"ContainerDied","Data":"573a3995683d82b1906fe90266ed653a442397cfb5dabf1e775694e4a8554fda"} Oct 30 20:12:07.828472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.828399 5991 scope.go:122] "RemoveContainer" containerID="231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527" Oct 30 20:12:07.840739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.840720 5991 scope.go:122] "RemoveContainer" containerID="231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527" Oct 30 20:12:07.841100 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:12:07.841080 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527\": container with ID starting with 231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527 not found: ID does not exist" containerID="231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527" Oct 30 20:12:07.841150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.841110 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527"} err="failed to get container status \"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527\": rpc error: code = NotFound desc = could not find container \"231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527\": container with ID starting with 231f4b039da585d7dc1bb02b4a35623a385ef4e99caf8049754a085032e5f527 not found: ID does not exist" Oct 30 20:12:07.890809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.890766 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:12:07.902142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:07.902106 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef132f-6bc59cd65d-vk7k5"] Oct 30 20:12:08.681822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:08.681786 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50d6d58-4921-4961-b36c-a2158faa1b26" path="/var/lib/kubelet/pods/a50d6d58-4921-4961-b36c-a2158faa1b26/volumes" Oct 30 20:12:09.448158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:09.448066 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:19.447588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:19.447545 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:20.872599 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:12:20.872557 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:12:27.373467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.373431 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:12:27.476282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.476241 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\" (UniqueName: \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\") pod \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " Oct 30 20:12:27.476519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.476475 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\") pod \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\" (UID: \"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa\") " Oct 30 20:12:27.476972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.476937 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume" pod "2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" (UID: "2b5e934b-9f7b-4ded-b54b-88c43a11a3fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:27.477325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.477303 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:12:27.479199 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.479165 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5" pod "2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" (UID: "2b5e934b-9f7b-4ded-b54b-88c43a11a3fa"). InnerVolumeSpecName "kube-api-access-r5tq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:27.579014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:27.578979 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tq5\" (UniqueName: \"kubernetes.io/projected/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa-kube-api-access-r5tq5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:12:28.003247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.003214 5991 generic.go:356] "Generic (PLEG): container finished" podID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerID="582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e" exitCode=0 Oct 30 20:12:28.003523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.003267 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" Oct 30 20:12:28.003523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.003295 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" event={"ID":"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa","Type":"ContainerDied","Data":"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e"} Oct 30 20:12:28.003523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.003354 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2" event={"ID":"2b5e934b-9f7b-4ded-b54b-88c43a11a3fa","Type":"ContainerDied","Data":"0242c7d9dc24f245b6829a7ffdc7c311052881f315eecf26727f30895a4358c3"} Oct 30 20:12:28.003523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.003394 5991 scope.go:122] "RemoveContainer" containerID="582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e" Oct 30 20:12:28.015416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.015390 5991 scope.go:122] "RemoveContainer" containerID="582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e" Oct 30 20:12:28.015761 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:12:28.015735 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e\": container with ID starting with 582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e not found: ID does not exist" containerID="582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e" Oct 30 20:12:28.015851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.015769 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e"} err="failed to get container status \"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e\": rpc error: code = NotFound desc = could not find container \"582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e\": container with ID starting with 582a3938bded0923867ececf3a79db740d0d8a3b24b95d6509e485b3209b555e not found: ID does not exist" Oct 30 20:12:28.072234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.072199 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:12:28.087031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.086990 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef132f-5687cfc75-tvpj2"] Oct 30 20:12:28.680938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:12:28.680893 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" path="/var/lib/kubelet/pods/2b5e934b-9f7b-4ded-b54b-88c43a11a3fa/volumes" Oct 30 20:12:50.887134 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:12:50.887081 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:13:15.550871 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:15.550785 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:13:20.875551 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:13:20.875511 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:13:50.873025 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:13:50.872984 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:13:51.578955 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:51.578910 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:13:51.579227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:51.579192 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" podUID="c7091269-3876-4331-9bb8-4b2a2c73bf18" containerName="maas-api" containerID="cri-o://3355c54095ef2aa2be82370faac5b4736d34c5f83349444928b2d7e85f1f68b4" gracePeriod=30 Oct 30 20:13:51.772016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:51.771973 5991 generic.go:356] "Generic (PLEG): container finished" podID="c7091269-3876-4331-9bb8-4b2a2c73bf18" containerID="3355c54095ef2aa2be82370faac5b4736d34c5f83349444928b2d7e85f1f68b4" exitCode=0 Oct 30 20:13:51.772357 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:51.772318 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" event={"ID":"c7091269-3876-4331-9bb8-4b2a2c73bf18","Type":"ContainerDied","Data":"3355c54095ef2aa2be82370faac5b4736d34c5f83349444928b2d7e85f1f68b4"} Oct 30 20:13:51.997445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:51.997407 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.078859 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\") pod \"c7091269-3876-4331-9bb8-4b2a2c73bf18\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.078912 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\") pod \"c7091269-3876-4331-9bb8-4b2a2c73bf18\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.078948 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") pod \"c7091269-3876-4331-9bb8-4b2a2c73bf18\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.079272 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\" (UniqueName: \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\") pod \"c7091269-3876-4331-9bb8-4b2a2c73bf18\" (UID: \"c7091269-3876-4331-9bb8-4b2a2c73bf18\") " Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.079838 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca" pod "c7091269-3876-4331-9bb8-4b2a2c73bf18" (UID: "c7091269-3876-4331-9bb8-4b2a2c73bf18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:52.082424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.080264 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7091269-3876-4331-9bb8-4b2a2c73bf18-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.129389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.115618 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m" pod "c7091269-3876-4331-9bb8-4b2a2c73bf18" (UID: "c7091269-3876-4331-9bb8-4b2a2c73bf18"). InnerVolumeSpecName "kube-api-access-fs76m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:52.145585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.138520 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls" pod "c7091269-3876-4331-9bb8-4b2a2c73bf18" (UID: "c7091269-3876-4331-9bb8-4b2a2c73bf18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:52.145585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.138611 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs" pod "c7091269-3876-4331-9bb8-4b2a2c73bf18" (UID: "c7091269-3876-4331-9bb8-4b2a2c73bf18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:52.185291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.185183 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs76m\" (UniqueName: \"kubernetes.io/projected/c7091269-3876-4331-9bb8-4b2a2c73bf18-kube-api-access-fs76m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.185291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.185227 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.185291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.185242 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7091269-3876-4331-9bb8-4b2a2c73bf18-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.784326 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.784295 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" Oct 30 20:13:52.784530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.784302 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f" event={"ID":"c7091269-3876-4331-9bb8-4b2a2c73bf18","Type":"ContainerDied","Data":"ae33a2d13c8e2a6d1d1f5f013f2c066cc519ef2be3c85ea63d379f820f32efdd"} Oct 30 20:13:52.784530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.784439 5991 scope.go:122] "RemoveContainer" containerID="3355c54095ef2aa2be82370faac5b4736d34c5f83349444928b2d7e85f1f68b4" Oct 30 20:13:52.840037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.840001 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:13:52.851944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:52.851908 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-936f70-7d8df79f6-k982f"] Oct 30 20:13:54.686626 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:54.686580 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7091269-3876-4331-9bb8-4b2a2c73bf18" path="/var/lib/kubelet/pods/c7091269-3876-4331-9bb8-4b2a2c73bf18/volumes" Oct 30 20:13:57.218626 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:57.218582 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 20:13:57.219191 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:57.218819 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" containerID="cri-o://8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e" gracePeriod=30 Oct 30 20:13:59.649395 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:59.646994 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:13:59.649395 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:13:59.647321 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" containerID="cri-o://bb496389bc3c225ca24c9d75bca883d90f7ca399d11a4cf1640cbbe3d4831d86" gracePeriod=30 Oct 30 20:14:01.673274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.673231 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 20:14:01.673780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.673467 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" containerID="cri-o://5492c66bb865df411a85676ee9f3ad717ff4d17d02a6d795d6352fc693bc598c" gracePeriod=30 Oct 30 20:14:01.939820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.939736 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9xsqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:01.939983 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.939820 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:01.959598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.959545 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 20:14:01.959828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:01.959787 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" containerID="cri-o://d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332" gracePeriod=30 Oct 30 20:14:02.523418 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.523389 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695561 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695665 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695691 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695749 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695791 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.695917 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.696009 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.696123 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.696218 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\") pod \"1e969921-f44b-49eb-b81b-fb3686f1d130\" (UID: \"1e969921-f44b-49eb-b81b-fb3686f1d130\") " Oct 30 20:14:02.697026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.696691 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.698072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.697101 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.698072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.697436 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.698072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.697442 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.698072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.697476 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.698072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.697774 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.698443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.698413 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.714448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.700141 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.714448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.700587 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:02.714448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.700922 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "kube-api-access-f9bcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.714448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.705754 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy" pod "1e969921-f44b-49eb-b81b-fb3686f1d130" (UID: "1e969921-f44b-49eb-b81b-fb3686f1d130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799479 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799515 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e969921-f44b-49eb-b81b-fb3686f1d130-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799528 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799540 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799555 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799567 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bcf\" (UniqueName: \"kubernetes.io/projected/1e969921-f44b-49eb-b81b-fb3686f1d130-kube-api-access-f9bcf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.799585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.799580 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e969921-f44b-49eb-b81b-fb3686f1d130-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.906196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.906159 5991 generic.go:356] "Generic (PLEG): container finished" podID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerID="8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e" exitCode=0 Oct 30 20:14:02.906421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.906232 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" Oct 30 20:14:02.906421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.906241 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" event={"ID":"1e969921-f44b-49eb-b81b-fb3686f1d130","Type":"ContainerDied","Data":"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e"} Oct 30 20:14:02.906421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.906307 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp" event={"ID":"1e969921-f44b-49eb-b81b-fb3686f1d130","Type":"ContainerDied","Data":"5a1cd732c4337a12014c31bd199afada54e79a5c62c96f5de6a594730f15a7b2"} Oct 30 20:14:02.906421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.906330 5991 scope.go:122] "RemoveContainer" containerID="8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e" Oct 30 20:14:02.918993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.918966 5991 scope.go:122] "RemoveContainer" containerID="8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e" Oct 30 20:14:02.919331 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:02.919311 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e\": container with ID starting with 8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e not found: ID does not exist" containerID="8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e" Oct 30 20:14:02.919450 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.919342 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e"} err="failed to get container status \"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e\": rpc error: code = NotFound desc = could not find container \"8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e\": container with ID starting with 8d7ec47692c3cf8a868045eeb2607f4a6c0a82a455446cba534abeeaa42b875e not found: ID does not exist" Oct 30 20:14:02.963142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.963109 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 20:14:02.979959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:02.978190 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9xsqp"] Oct 30 20:14:03.905448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:03.905407 5991 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:03.905855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:03.905470 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:04.681331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.681293 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" path="/var/lib/kubelet/pods/1e969921-f44b-49eb-b81b-fb3686f1d130/volumes" Oct 30 20:14:04.925088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.925054 5991 generic.go:356] "Generic (PLEG): container finished" podID="dcfd2117-344b-4f12-95d3-242553727ad3" containerID="bb496389bc3c225ca24c9d75bca883d90f7ca399d11a4cf1640cbbe3d4831d86" exitCode=0 Oct 30 20:14:04.925507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.925136 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" event={"ID":"dcfd2117-344b-4f12-95d3-242553727ad3","Type":"ContainerDied","Data":"bb496389bc3c225ca24c9d75bca883d90f7ca399d11a4cf1640cbbe3d4831d86"} Oct 30 20:14:04.925507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.925200 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" event={"ID":"dcfd2117-344b-4f12-95d3-242553727ad3","Type":"ContainerDied","Data":"cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5"} Oct 30 20:14:04.925507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.925226 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbcde2616274f1a6a4e818b73571fd6875fa7876a0922f96b1dcac8173b909b5" Oct 30 20:14:04.942513 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:04.942481 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:14:05.045257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.045204 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.045257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.045263 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.045581 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.045308 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.045647 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.045583 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:05.045900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.045874 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.046161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.046136 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.046709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.046561 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:05.046709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.046694 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.047021 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.046997 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:05.047224 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.047205 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.047716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.047678 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.047990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.047965 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:05.048249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.048099 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\") pod \"dcfd2117-344b-4f12-95d3-242553727ad3\" (UID: \"dcfd2117-344b-4f12-95d3-242553727ad3\") " Oct 30 20:14:05.048249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.048200 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:05.048429 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.048357 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:05.048915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.048890 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:05.049083 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049045 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:05.049501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049481 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfd2117-344b-4f12-95d3-242553727ad3-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049506 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049520 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049533 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049546 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049561 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfd2117-344b-4f12-95d3-242553727ad3-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049574 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfd2117-344b-4f12-95d3-242553727ad3-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.049901 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.049586 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.050142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.050118 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp" pod "dcfd2117-344b-4f12-95d3-242553727ad3" (UID: "dcfd2117-344b-4f12-95d3-242553727ad3"). InnerVolumeSpecName "kube-api-access-f25sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:05.151438 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.151402 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25sp\" (UniqueName: \"kubernetes.io/projected/dcfd2117-344b-4f12-95d3-242553727ad3-kube-api-access-f25sp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:05.803032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.802987 5991 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:05.803252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.803065 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:05.934186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:05.934147 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh" Oct 30 20:14:06.000396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.000343 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:14:06.012706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.012667 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hwvfh"] Oct 30 20:14:06.683164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.683127 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" path="/var/lib/kubelet/pods/dcfd2117-344b-4f12-95d3-242553727ad3/volumes" Oct 30 20:14:06.944889 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.944863 5991 generic.go:356] "Generic (PLEG): container finished" podID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerID="5492c66bb865df411a85676ee9f3ad717ff4d17d02a6d795d6352fc693bc598c" exitCode=0 Oct 30 20:14:06.945181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.944936 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" event={"ID":"3c55fa64-bfb5-44ff-a606-703ffb19007a","Type":"ContainerDied","Data":"5492c66bb865df411a85676ee9f3ad717ff4d17d02a6d795d6352fc693bc598c"} Oct 30 20:14:06.945181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.945019 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" event={"ID":"3c55fa64-bfb5-44ff-a606-703ffb19007a","Type":"ContainerDied","Data":"b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8"} Oct 30 20:14:06.945181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.945038 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b82d4d882bc1275648f4d8f78df782d1e11f9fc1637a4744bb19f526b32f49a8" Oct 30 20:14:06.975306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:06.975270 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 20:14:07.084084 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084042 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084161 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084216 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084259 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084327 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084393 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084406 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.084631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084421 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084570 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.084900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084629 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.084900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.084691 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\") pod \"3c55fa64-bfb5-44ff-a606-703ffb19007a\" (UID: \"3c55fa64-bfb5-44ff-a606-703ffb19007a\") " Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085179 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085577 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085600 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085615 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085795 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.086034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.085990 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:07.092136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.092094 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "kube-api-access-82rrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:07.092136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.092099 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:07.092328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.092145 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:07.092479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.092454 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy" pod "3c55fa64-bfb5-44ff-a606-703ffb19007a" (UID: "3c55fa64-bfb5-44ff-a606-703ffb19007a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.189050 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189012 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.189050 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189050 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c55fa64-bfb5-44ff-a606-703ffb19007a-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.189330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189063 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.189330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189076 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.189330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189091 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rrh\" (UniqueName: \"kubernetes.io/projected/3c55fa64-bfb5-44ff-a606-703ffb19007a-kube-api-access-82rrh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.189330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.189106 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c55fa64-bfb5-44ff-a606-703ffb19007a-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.280757 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.280728 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 20:14:07.392241 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392205 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392241 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392248 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392283 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392347 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392515 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392578 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392598 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392660 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392682 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392841 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.392899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.392877 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\") pod \"a63b3b88-539f-4c10-859a-904431939bb7\" (UID: \"a63b3b88-539f-4c10-859a-904431939bb7\") " Oct 30 20:14:07.393282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.393260 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:07.393521 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.393492 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.393713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.393690 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.393997 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.393980 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.394036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394003 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63b3b88-539f-4c10-859a-904431939bb7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.394036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394020 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.394036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394032 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.394135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394045 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.394852 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394825 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:07.394938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.394887 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "kube-api-access-5bqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:07.395028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.395011 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:07.395088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.395049 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token" pod "a63b3b88-539f-4c10-859a-904431939bb7" (UID: "a63b3b88-539f-4c10-859a-904431939bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:07.495201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.495152 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63b3b88-539f-4c10-859a-904431939bb7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.495201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.495192 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.495201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.495204 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bqmd\" (UniqueName: \"kubernetes.io/projected/a63b3b88-539f-4c10-859a-904431939bb7-kube-api-access-5bqmd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.495201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.495214 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63b3b88-539f-4c10-859a-904431939bb7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:07.955548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955445 5991 generic.go:356] "Generic (PLEG): container finished" podID="a63b3b88-539f-4c10-859a-904431939bb7" containerID="d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332" exitCode=0 Oct 30 20:14:07.955548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955519 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" Oct 30 20:14:07.956051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955541 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" event={"ID":"a63b3b88-539f-4c10-859a-904431939bb7","Type":"ContainerDied","Data":"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332"} Oct 30 20:14:07.956051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955596 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2" event={"ID":"a63b3b88-539f-4c10-859a-904431939bb7","Type":"ContainerDied","Data":"91bedb38fb1c988a77a2befee897222979f4d3f398244dd1ef7b2f46e00adfde"} Oct 30 20:14:07.956051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955619 5991 scope.go:122] "RemoveContainer" containerID="d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332" Oct 30 20:14:07.956051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.955831 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql" Oct 30 20:14:07.973554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.973533 5991 scope.go:122] "RemoveContainer" containerID="d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332" Oct 30 20:14:07.973884 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:07.973864 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332\": container with ID starting with d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332 not found: ID does not exist" containerID="d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332" Oct 30 20:14:07.973960 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:07.973891 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332"} err="failed to get container status \"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332\": rpc error: code = NotFound desc = could not find container \"d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332\": container with ID starting with d70a1420b04f940e3e4b9743743262fe7d2782cbbd552254dbd0a366c15e1332 not found: ID does not exist" Oct 30 20:14:08.011607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.011566 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 20:14:08.023266 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.023227 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t5cql"] Oct 30 20:14:08.049634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.049584 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 20:14:08.053459 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.053426 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-drcg2"] Oct 30 20:14:08.681399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.681328 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" path="/var/lib/kubelet/pods/3c55fa64-bfb5-44ff-a606-703ffb19007a/volumes" Oct 30 20:14:08.681938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:08.681922 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63b3b88-539f-4c10-859a-904431939bb7" path="/var/lib/kubelet/pods/a63b3b88-539f-4c10-859a-904431939bb7/volumes" Oct 30 20:14:19.778445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:19.778389 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:14:20.899680 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:20.899592 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:14:26.568508 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:26.568471 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:14:26.600678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:26.600634 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:14:29.725951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:29.725912 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:14:31.569072 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:31.569024 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" podUID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" containerName="payload-processing" containerID="cri-o://45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107" gracePeriod=25 Oct 30 20:14:31.601992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:31.601172 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" containerID="cri-o://4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7" gracePeriod=25 Oct 30 20:14:31.905683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:31.905654 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:14:32.025016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.024976 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\") pod \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " Oct 30 20:14:32.025251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.025044 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\" (UniqueName: \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\") pod \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\" (UID: \"0e86c9aa-1614-49a8-8008-95bdd7f2b77b\") " Oct 30 20:14:32.025462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.025384 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume" pod "0e86c9aa-1614-49a8-8008-95bdd7f2b77b" (UID: "0e86c9aa-1614-49a8-8008-95bdd7f2b77b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:32.025736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.025714 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:32.027838 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.027811 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb" pod "0e86c9aa-1614-49a8-8008-95bdd7f2b77b" (UID: "0e86c9aa-1614-49a8-8008-95bdd7f2b77b"). InnerVolumeSpecName "kube-api-access-qncmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:32.127114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.127007 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncmb\" (UniqueName: \"kubernetes.io/projected/0e86c9aa-1614-49a8-8008-95bdd7f2b77b-kube-api-access-qncmb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:32.169436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.169381 5991 generic.go:356] "Generic (PLEG): container finished" podID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" containerID="45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107" exitCode=0 Oct 30 20:14:32.169618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.169443 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" event={"ID":"0e86c9aa-1614-49a8-8008-95bdd7f2b77b","Type":"ContainerDied","Data":"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107"} Oct 30 20:14:32.169618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.169464 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" Oct 30 20:14:32.169618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.169514 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt" event={"ID":"0e86c9aa-1614-49a8-8008-95bdd7f2b77b","Type":"ContainerDied","Data":"94525d371ca204b0b58db58d7590389cfbd5c50be367769204aa6350e1064536"} Oct 30 20:14:32.169618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.169549 5991 scope.go:122] "RemoveContainer" containerID="45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107" Oct 30 20:14:32.183705 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.183674 5991 scope.go:122] "RemoveContainer" containerID="45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107" Oct 30 20:14:32.184078 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:32.184051 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107\": container with ID starting with 45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107 not found: ID does not exist" containerID="45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107" Oct 30 20:14:32.184138 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.184093 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107"} err="failed to get container status \"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107\": rpc error: code = NotFound desc = could not find container \"45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107\": container with ID starting with 45ff87f7a435a4daff0589028e7f6751353047af1abef454e475d001c28a8107 not found: ID does not exist" Oct 30 20:14:32.233339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.233300 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:14:32.237234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.237200 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebe57b-5576bcc7f6-blhqt"] Oct 30 20:14:32.347990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.347953 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 20:14:32.348193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.348171 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" containerID="cri-o://d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd" gracePeriod=30 Oct 30 20:14:32.681873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:32.681835 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" path="/var/lib/kubelet/pods/0e86c9aa-1614-49a8-8008-95bdd7f2b77b/volumes" Oct 30 20:14:34.144869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:34.144833 5991 scope.go:122] "RemoveContainer" containerID="5492c66bb865df411a85676ee9f3ad717ff4d17d02a6d795d6352fc693bc598c" Oct 30 20:14:34.726222 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:34.726170 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" containerID="cri-o://8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000" gracePeriod=25 Oct 30 20:14:37.670066 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.670036 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 20:14:37.717638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717604 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717672 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717722 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717779 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717955 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717872 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717955 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717896 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.717955 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717916 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:37.718131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.717978 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.718131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718087 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.718237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718163 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:37.718237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718227 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\") pod \"c25a782d-e324-4336-b93d-bc292e630f5e\" (UID: \"c25a782d-e324-4336-b93d-bc292e630f5e\") " Oct 30 20:14:37.718344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718258 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:37.718344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718158 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:37.718620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.718595 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:37.719192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.719168 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.719192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.719190 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.719424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.719199 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.719424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.719210 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.719424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.719222 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25a782d-e324-4336-b93d-bc292e630f5e-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.720213 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.720191 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:37.720301 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.720278 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:37.720571 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.720557 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:37.720686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.720673 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq" pod "c25a782d-e324-4336-b93d-bc292e630f5e" (UID: "c25a782d-e324-4336-b93d-bc292e630f5e"). InnerVolumeSpecName "kube-api-access-zw8nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:37.821308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.821265 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8nq\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-kube-api-access-zw8nq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.821308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.821301 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25a782d-e324-4336-b93d-bc292e630f5e-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.821308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.821313 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25a782d-e324-4336-b93d-bc292e630f5e-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.821308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:37.821320 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25a782d-e324-4336-b93d-bc292e630f5e-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:38.222846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.222807 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:38.238215 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.238175 5991 generic.go:356] "Generic (PLEG): container finished" podID="c25a782d-e324-4336-b93d-bc292e630f5e" containerID="d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd" exitCode=0 Oct 30 20:14:38.238416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.238221 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" Oct 30 20:14:38.238416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.238249 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerDied","Data":"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd"} Oct 30 20:14:38.238416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.238315 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5" event={"ID":"c25a782d-e324-4336-b93d-bc292e630f5e","Type":"ContainerDied","Data":"03bb95ad3eec823bc8a5e67b2c7292cd5b33cbe319baa76c5250e808c0176628"} Oct 30 20:14:38.238416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.238350 5991 scope.go:122] "RemoveContainer" containerID="d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd" Oct 30 20:14:38.252204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.252179 5991 scope.go:122] "RemoveContainer" containerID="2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14" Oct 30 20:14:38.263634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.263611 5991 scope.go:122] "RemoveContainer" containerID="d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd" Oct 30 20:14:38.264013 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:38.263989 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd\": container with ID starting with d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd not found: ID does not exist" containerID="d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd" Oct 30 20:14:38.264078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.264022 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd"} err="failed to get container status \"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd\": rpc error: code = NotFound desc = could not find container \"d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd\": container with ID starting with d9fb436e1d6dd87014c7aa21110d6ba0e4a1701a74cf4d9358e7eb3b6c7a91cd not found: ID does not exist" Oct 30 20:14:38.264078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.264049 5991 scope.go:122] "RemoveContainer" containerID="2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14" Oct 30 20:14:38.264356 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:38.264334 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14\": container with ID starting with 2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14 not found: ID does not exist" containerID="2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14" Oct 30 20:14:38.264356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.264377 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14"} err="failed to get container status \"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14\": rpc error: code = NotFound desc = could not find container \"2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14\": container with ID starting with 2ca76cbcb6bff9fda90749b936458e3749539b8502200861fb9c347514918d14 not found: ID does not exist" Oct 30 20:14:38.291142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.291102 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 20:14:38.304235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.304194 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-85nd5"] Oct 30 20:14:38.683886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:38.683789 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" path="/var/lib/kubelet/pods/c25a782d-e324-4336-b93d-bc292e630f5e/volumes" Oct 30 20:14:40.530950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:40.530910 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:48.223150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:48.223058 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:50.530611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:50.530572 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:50.873438 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:50.873318 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:14:51.902879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.902848 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:14:51.960932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.960894 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\") pod \"1c863987-0379-4189-9e5e-46836126458d\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " Oct 30 20:14:51.961344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.961320 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume" pod "1c863987-0379-4189-9e5e-46836126458d" (UID: "1c863987-0379-4189-9e5e-46836126458d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:51.961511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.961480 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\" (UniqueName: \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\") pod \"1c863987-0379-4189-9e5e-46836126458d\" (UID: \"1c863987-0379-4189-9e5e-46836126458d\") " Oct 30 20:14:51.962522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.962498 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c863987-0379-4189-9e5e-46836126458d-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:51.964175 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:51.964148 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p" pod "1c863987-0379-4189-9e5e-46836126458d" (UID: "1c863987-0379-4189-9e5e-46836126458d"). InnerVolumeSpecName "kube-api-access-xqj5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:52.064010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.063975 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqj5p\" (UniqueName: \"kubernetes.io/projected/1c863987-0379-4189-9e5e-46836126458d-kube-api-access-xqj5p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:52.358678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.358627 5991 generic.go:356] "Generic (PLEG): container finished" podID="1c863987-0379-4189-9e5e-46836126458d" containerID="4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7" exitCode=0 Oct 30 20:14:52.358863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.358695 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" Oct 30 20:14:52.358863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.358708 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" event={"ID":"1c863987-0379-4189-9e5e-46836126458d","Type":"ContainerDied","Data":"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7"} Oct 30 20:14:52.358863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.358767 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9" event={"ID":"1c863987-0379-4189-9e5e-46836126458d","Type":"ContainerDied","Data":"085ddf8da6ff3d57e5366b5a86cbe710e974e2188da0df8b65492391305994dc"} Oct 30 20:14:52.358863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.358799 5991 scope.go:122] "RemoveContainer" containerID="4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7" Oct 30 20:14:52.370732 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.370706 5991 scope.go:122] "RemoveContainer" containerID="4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7" Oct 30 20:14:52.370993 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:52.370971 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7\": container with ID starting with 4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7 not found: ID does not exist" containerID="4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7" Oct 30 20:14:52.371056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.371000 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7"} err="failed to get container status \"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7\": rpc error: code = NotFound desc = could not find container \"4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7\": container with ID starting with 4fafb5c515b31bbc0c29ee8b765c722c4500a0138db57e346e60bb921b185ea7 not found: ID does not exist" Oct 30 20:14:52.412875 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.412836 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:14:52.425062 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.425024 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebe57b-b8f7f8dff-8s4b9"] Oct 30 20:14:52.692634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:52.692557 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c863987-0379-4189-9e5e-46836126458d" path="/var/lib/kubelet/pods/1c863987-0379-4189-9e5e-46836126458d/volumes" Oct 30 20:14:55.031735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.031705 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:14:55.111119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.111080 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\" (UniqueName: \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\") pod \"918edd0c-ea25-4710-8d80-2ee6616501a9\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " Oct 30 20:14:55.111465 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.111442 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\") pod \"918edd0c-ea25-4710-8d80-2ee6616501a9\" (UID: \"918edd0c-ea25-4710-8d80-2ee6616501a9\") " Oct 30 20:14:55.111812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.111788 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume" pod "918edd0c-ea25-4710-8d80-2ee6616501a9" (UID: "918edd0c-ea25-4710-8d80-2ee6616501a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:55.112173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.112154 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918edd0c-ea25-4710-8d80-2ee6616501a9-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:55.113445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.113415 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg" pod "918edd0c-ea25-4710-8d80-2ee6616501a9" (UID: "918edd0c-ea25-4710-8d80-2ee6616501a9"). InnerVolumeSpecName "kube-api-access-whwbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:55.213627 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.213590 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whwbg\" (UniqueName: \"kubernetes.io/projected/918edd0c-ea25-4710-8d80-2ee6616501a9-kube-api-access-whwbg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:55.385962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.385872 5991 generic.go:356] "Generic (PLEG): container finished" podID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerID="8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000" exitCode=0 Oct 30 20:14:55.385962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.385919 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" Oct 30 20:14:55.386207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.385956 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" event={"ID":"918edd0c-ea25-4710-8d80-2ee6616501a9","Type":"ContainerDied","Data":"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000"} Oct 30 20:14:55.386207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.386016 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl" event={"ID":"918edd0c-ea25-4710-8d80-2ee6616501a9","Type":"ContainerDied","Data":"2c660c16e9111c4bcf4631ae0138151f38a86eb3d474049fd17ceefb61c5f472"} Oct 30 20:14:55.386207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.386045 5991 scope.go:122] "RemoveContainer" containerID="8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000" Oct 30 20:14:55.398050 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.398028 5991 scope.go:122] "RemoveContainer" containerID="8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000" Oct 30 20:14:55.398441 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:14:55.398413 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000\": container with ID starting with 8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000 not found: ID does not exist" containerID="8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000" Oct 30 20:14:55.398488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.398456 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000"} err="failed to get container status \"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000\": rpc error: code = NotFound desc = could not find container \"8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000\": container with ID starting with 8cfdcf275d237c4351d3736d012592092e0b9f130035f2e7410d0d7994fd6000 not found: ID does not exist" Oct 30 20:14:55.439043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.439005 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:14:55.451420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:55.451387 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b5635-6657f7c79-9fpjl"] Oct 30 20:14:56.680842 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:56.680801 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" path="/var/lib/kubelet/pods/918edd0c-ea25-4710-8d80-2ee6616501a9/volumes" Oct 30 20:14:57.723488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:57.723451 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:14:57.724186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:57.723905 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" containerID="cri-o://035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809" gracePeriod=30 Oct 30 20:14:59.347423 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.347387 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:14:59.348052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.347611 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" podUID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" containerName="maas-api" containerID="cri-o://cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16" gracePeriod=30 Oct 30 20:14:59.670988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.670958 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:14:59.822990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.822945 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\") pod \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " Oct 30 20:14:59.823184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.823073 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\" (UniqueName: \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\") pod \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " Oct 30 20:14:59.823350 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.823331 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\") pod \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " Oct 30 20:14:59.823442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.823424 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\") pod \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\" (UID: \"807ddeb4-5347-4f20-8f10-5e800bd0f2df\") " Oct 30 20:14:59.823828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.823804 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca" pod "807ddeb4-5347-4f20-8f10-5e800bd0f2df" (UID: "807ddeb4-5347-4f20-8f10-5e800bd0f2df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:59.824151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.824132 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807ddeb4-5347-4f20-8f10-5e800bd0f2df-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:59.825347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.825319 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4" pod "807ddeb4-5347-4f20-8f10-5e800bd0f2df" (UID: "807ddeb4-5347-4f20-8f10-5e800bd0f2df"). InnerVolumeSpecName "kube-api-access-w24k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:59.825347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.825338 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs" pod "807ddeb4-5347-4f20-8f10-5e800bd0f2df" (UID: "807ddeb4-5347-4f20-8f10-5e800bd0f2df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:59.825620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.825356 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls" pod "807ddeb4-5347-4f20-8f10-5e800bd0f2df" (UID: "807ddeb4-5347-4f20-8f10-5e800bd0f2df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:59.926283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.926048 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:59.926283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.926127 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807ddeb4-5347-4f20-8f10-5e800bd0f2df-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:14:59.926740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:14:59.926689 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24k4\" (UniqueName: \"kubernetes.io/projected/807ddeb4-5347-4f20-8f10-5e800bd0f2df-kube-api-access-w24k4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:00.265735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.265692 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:00.267060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267029 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" containerName="payload-processing" Oct 30 20:15:00.267060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267057 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" containerName="payload-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267073 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a50d6d58-4921-4961-b36c-a2158faa1b26" containerName="payload-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267080 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50d6d58-4921-4961-b36c-a2158faa1b26" containerName="payload-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267107 5991 cpu_manager.go:414] "containerMap: removing container" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267114 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267132 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267139 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267155 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267162 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267174 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267181 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267193 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267200 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267209 5991 cpu_manager.go:414] "containerMap: removing container" podUID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" containerName="maas-api" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267217 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" containerName="maas-api" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267231 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267238 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267254 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267261 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.267271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267280 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c7091269-3876-4331-9bb8-4b2a2c73bf18" containerName="maas-api" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267287 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7091269-3876-4331-9bb8-4b2a2c73bf18" containerName="maas-api" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267302 5991 cpu_manager.go:414] "containerMap: removing container" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267311 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267344 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267351 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267558 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c55fa64-bfb5-44ff-a606-703ffb19007a" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267578 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50d6d58-4921-4961-b36c-a2158faa1b26" containerName="payload-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267591 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e86c9aa-1614-49a8-8008-95bdd7f2b77b" containerName="payload-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267604 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfd2117-344b-4f12-95d3-242553727ad3" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267616 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63b3b88-539f-4c10-859a-904431939bb7" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267626 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e969921-f44b-49eb-b81b-fb3686f1d130" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267638 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5e934b-9f7b-4ded-b54b-88c43a11a3fa" containerName="payload-pre-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267647 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c863987-0379-4189-9e5e-46836126458d" containerName="payload-pre-processing" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267657 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" containerName="maas-api" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267665 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267674 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25a782d-e324-4336-b93d-bc292e630f5e" containerName="istio-proxy" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267684 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7091269-3876-4331-9bb8-4b2a2c73bf18" containerName="maas-api" Oct 30 20:15:00.268227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.267693 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="918edd0c-ea25-4710-8d80-2ee6616501a9" containerName="payload-pre-processing" Oct 30 20:15:00.272280 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.272256 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.279080 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.279048 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:00.284281 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.284258 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.327442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.315482 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:00.334341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.334304 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.335462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.335087 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:15:00.355734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.355685 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:00.357341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.357307 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4bz\" (UniqueName: \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.357610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.357576 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.365481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.365448 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:00.381008 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.380967 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:00.430404 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.430341 5991 generic.go:356] "Generic (PLEG): container finished" podID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" containerID="cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16" exitCode=0 Oct 30 20:15:00.430620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.430428 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" event={"ID":"807ddeb4-5347-4f20-8f10-5e800bd0f2df","Type":"ContainerDied","Data":"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16"} Oct 30 20:15:00.430620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.430455 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" Oct 30 20:15:00.430620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.430491 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv" event={"ID":"807ddeb4-5347-4f20-8f10-5e800bd0f2df","Type":"ContainerDied","Data":"cc4738540fe9003cca26d7d9e3a7453247e37dfd210947832fe3f736695bbe7b"} Oct 30 20:15:00.430620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.430518 5991 scope.go:122] "RemoveContainer" containerID="cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16" Oct 30 20:15:00.444602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.443573 5991 scope.go:122] "RemoveContainer" containerID="cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16" Oct 30 20:15:00.446220 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:00.445821 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16\": container with ID starting with cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16 not found: ID does not exist" containerID="cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16" Oct 30 20:15:00.446220 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.445869 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16"} err="failed to get container status \"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16\": rpc error: code = NotFound desc = could not find container \"cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16\": container with ID starting with cdeab24ff433631b36b98c9220e34a2c3d4d25833cc4c3d0cff278df87827c16 not found: ID does not exist" Oct 30 20:15:00.460188 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460153 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rth4l\" (UniqueName: \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.460422 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460332 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.460483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460421 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4bz\" (UniqueName: \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.460532 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460485 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.460664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460639 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.460728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.460709 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7mpv\" (UniqueName: \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.461303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.461286 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.495648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.495615 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4bz\" (UniqueName: \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.523271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.523186 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:15:00.526890 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.526864 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17154-6bc69fd59c-zbklv"] Oct 30 20:15:00.561973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.561938 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.562190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.562112 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.562190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.562175 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7mpv\" (UniqueName: \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.562296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.562247 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rth4l\" (UniqueName: \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.562819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.562793 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.563012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.562891 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.585706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.585656 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rth4l\" (UniqueName: \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.585706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.585677 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7mpv\" (UniqueName: \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\") pod \"maas-api-key-cleanup-29889855-s5bjb\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.593190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.593157 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:00.629196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.629160 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:00.679077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.674502 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:00.710604 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.710546 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807ddeb4-5347-4f20-8f10-5e800bd0f2df" path="/var/lib/kubelet/pods/807ddeb4-5347-4f20-8f10-5e800bd0f2df/volumes" Oct 30 20:15:00.912693 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:15:00.912656 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad70126_a20e_4a9a_a96e_54765276567d.slice/crio-c52cc3b7406c63ba36f80f01eb4860fc5ff2ae007daa94d2400365ea65391d9f WatchSource:0}: Error finding container c52cc3b7406c63ba36f80f01eb4860fc5ff2ae007daa94d2400365ea65391d9f: Status 404 returned error can't find the container with id c52cc3b7406c63ba36f80f01eb4860fc5ff2ae007daa94d2400365ea65391d9f Oct 30 20:15:00.913104 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.913083 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:00.952761 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:00.952730 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:00.953725 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:15:00.953693 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad16761_0452_4aa7_a2b1_118c4062f370.slice/crio-3a6f898ad8ef096d577d3a60d8250038c61bbe83d0b3bb5f39dbb2d055e3d9b0 WatchSource:0}: Error finding container 3a6f898ad8ef096d577d3a60d8250038c61bbe83d0b3bb5f39dbb2d055e3d9b0: Status 404 returned error can't find the container with id 3a6f898ad8ef096d577d3a60d8250038c61bbe83d0b3bb5f39dbb2d055e3d9b0 Oct 30 20:15:01.011787 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.011756 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:01.012226 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:15:01.012198 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae9d66_d044_4547_b344_feba5cd776f1.slice/crio-85c2996875d904f74c4a26f857d7b9c84315c9c6fd7cabae3b8a76103cd02456 WatchSource:0}: Error finding container 85c2996875d904f74c4a26f857d7b9c84315c9c6fd7cabae3b8a76103cd02456: Status 404 returned error can't find the container with id 85c2996875d904f74c4a26f857d7b9c84315c9c6fd7cabae3b8a76103cd02456 Oct 30 20:15:01.389316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.389277 5991 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:01.389722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.389344 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:01.440872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.440832 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerStarted","Data":"3a6f898ad8ef096d577d3a60d8250038c61bbe83d0b3bb5f39dbb2d055e3d9b0"} Oct 30 20:15:01.442558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.442521 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerStarted","Data":"7f536938f430b03b744ebf45623ac8803d7bc0b2518b02b46368d1ce3bcb0d28"} Oct 30 20:15:01.442728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.442573 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerStarted","Data":"c52cc3b7406c63ba36f80f01eb4860fc5ff2ae007daa94d2400365ea65391d9f"} Oct 30 20:15:01.444226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.444199 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerStarted","Data":"d6a98c3d1fe1558d66b5e5c51311102caea9396fd88f0f7708c5b1363650ef94"} Oct 30 20:15:01.444419 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.444245 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerStarted","Data":"85c2996875d904f74c4a26f857d7b9c84315c9c6fd7cabae3b8a76103cd02456"} Oct 30 20:15:01.481714 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.481660 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" podStartSLOduration=1.481646383 podStartE2EDuration="1.481646383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.480501436 +0000 UTC m=+2368.929994161" watchObservedRunningTime="2026-10-30 20:15:01.481646383 +0000 UTC m=+2368.931139107" Oct 30 20:15:01.524343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:01.524270 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" podStartSLOduration=1.5242497739999998 podStartE2EDuration="1.524249774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.520668617 +0000 UTC m=+2368.970161343" watchObservedRunningTime="2026-10-30 20:15:01.524249774 +0000 UTC m=+2368.973742499" Oct 30 20:15:02.454724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.454688 5991 generic.go:356] "Generic (PLEG): container finished" podID="5cae9d66-d044-4547-b344-feba5cd776f1" containerID="d6a98c3d1fe1558d66b5e5c51311102caea9396fd88f0f7708c5b1363650ef94" exitCode=60 Oct 30 20:15:02.455245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.454815 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerDied","Data":"d6a98c3d1fe1558d66b5e5c51311102caea9396fd88f0f7708c5b1363650ef94"} Oct 30 20:15:02.455245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.455169 5991 scope.go:122] "RemoveContainer" containerID="d6a98c3d1fe1558d66b5e5c51311102caea9396fd88f0f7708c5b1363650ef94" Oct 30 20:15:02.456958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.456931 5991 generic.go:356] "Generic (PLEG): container finished" podID="fad16761-0452-4aa7-a2b1-118c4062f370" containerID="04c0563f86c1b8a10e55626b2d122d0460425182b16f5af3f2fd1f72ceb0dc8b" exitCode=60 Oct 30 20:15:02.457074 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.457014 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerDied","Data":"04c0563f86c1b8a10e55626b2d122d0460425182b16f5af3f2fd1f72ceb0dc8b"} Oct 30 20:15:02.457424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.457407 5991 scope.go:122] "RemoveContainer" containerID="04c0563f86c1b8a10e55626b2d122d0460425182b16f5af3f2fd1f72ceb0dc8b" Oct 30 20:15:02.459115 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.459086 5991 generic.go:356] "Generic (PLEG): container finished" podID="aad70126-a20e-4a9a-a96e-54765276567d" containerID="7f536938f430b03b744ebf45623ac8803d7bc0b2518b02b46368d1ce3bcb0d28" exitCode=60 Oct 30 20:15:02.459227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.459113 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerDied","Data":"7f536938f430b03b744ebf45623ac8803d7bc0b2518b02b46368d1ce3bcb0d28"} Oct 30 20:15:02.459555 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:02.459536 5991 scope.go:122] "RemoveContainer" containerID="7f536938f430b03b744ebf45623ac8803d7bc0b2518b02b46368d1ce3bcb0d28" Oct 30 20:15:03.106273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.106240 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:15:03.210918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.210884 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.210964 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211110 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211220 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211240 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:03.211299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211333 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211360 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211410 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211450 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\") pod \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\" (UID: \"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a\") " Oct 30 20:15:03.211553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211511 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:03.211923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.211887 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:03.212391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.212242 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:03.212391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.212277 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:03.212391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.212321 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.212391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.212342 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.212391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.212357 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.213965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.213942 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:03.214206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.214177 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:03.214524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.214507 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:03.214732 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.214717 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l" pod "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" (UID: "0123ae97-5cff-49f0-ab2b-c8261ef9bf7a"). InnerVolumeSpecName "kube-api-access-nbm2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313887 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313920 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313930 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313940 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313949 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbm2l\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-kube-api-access-nbm2l\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.313982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.313959 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:03.471898 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.471861 5991 generic.go:356] "Generic (PLEG): container finished" podID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerID="035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809" exitCode=0 Oct 30 20:15:03.472445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.471919 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" Oct 30 20:15:03.472445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.471972 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" event={"ID":"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a","Type":"ContainerDied","Data":"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809"} Oct 30 20:15:03.472445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.472265 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb" event={"ID":"0123ae97-5cff-49f0-ab2b-c8261ef9bf7a","Type":"ContainerDied","Data":"cc1a26818fdc017a293cd2ed7b55addba649cb8116af186b6e6dbdec952fbe11"} Oct 30 20:15:03.472445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.472419 5991 scope.go:122] "RemoveContainer" containerID="035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809" Oct 30 20:15:03.475206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.475173 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerStarted","Data":"522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f"} Oct 30 20:15:03.477586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.477554 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerStarted","Data":"820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55"} Oct 30 20:15:03.480075 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.480036 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerStarted","Data":"bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d"} Oct 30 20:15:03.486067 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.485908 5991 scope.go:122] "RemoveContainer" containerID="035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809" Oct 30 20:15:03.486263 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:03.486236 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809\": container with ID starting with 035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809 not found: ID does not exist" containerID="035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809" Oct 30 20:15:03.486305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.486273 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809"} err="failed to get container status \"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809\": rpc error: code = NotFound desc = could not find container \"035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809\": container with ID starting with 035378bb86fe1099d7a7a1ba9c987550fd71ded7e50b603c06cc5dcfe7be0809 not found: ID does not exist" Oct 30 20:15:03.531118 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.531082 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:15:03.556533 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.549952 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-5mrcb"] Oct 30 20:15:03.594727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:03.594593 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" podStartSLOduration=3.594573426 podStartE2EDuration="3.594573426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.592307575 +0000 UTC m=+2371.041800300" watchObservedRunningTime="2026-10-30 20:15:03.594573426 +0000 UTC m=+2371.044066154" Oct 30 20:15:04.493967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.493926 5991 generic.go:356] "Generic (PLEG): container finished" podID="fad16761-0452-4aa7-a2b1-118c4062f370" containerID="bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d" exitCode=60 Oct 30 20:15:04.494492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.494005 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerDied","Data":"bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d"} Oct 30 20:15:04.494492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.494099 5991 scope.go:122] "RemoveContainer" containerID="04c0563f86c1b8a10e55626b2d122d0460425182b16f5af3f2fd1f72ceb0dc8b" Oct 30 20:15:04.494492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.494454 5991 scope.go:122] "RemoveContainer" containerID="bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d" Oct 30 20:15:04.494873 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:04.494851 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-s5bjb_odh-ai-gateway-infra(fad16761-0452-4aa7-a2b1-118c4062f370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" Oct 30 20:15:04.497553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.497530 5991 generic.go:356] "Generic (PLEG): container finished" podID="aad70126-a20e-4a9a-a96e-54765276567d" containerID="522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f" exitCode=60 Oct 30 20:15:04.497667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.497610 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerDied","Data":"522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f"} Oct 30 20:15:04.498003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.497984 5991 scope.go:122] "RemoveContainer" containerID="522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f" Oct 30 20:15:04.498440 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:04.498401 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx_odh-ai-gateway-infra(aad70126-a20e-4a9a-a96e-54765276567d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" podUID="aad70126-a20e-4a9a-a96e-54765276567d" Oct 30 20:15:04.499727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.499703 5991 generic.go:356] "Generic (PLEG): container finished" podID="5cae9d66-d044-4547-b344-feba5cd776f1" containerID="820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55" exitCode=60 Oct 30 20:15:04.499809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.499746 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerDied","Data":"820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55"} Oct 30 20:15:04.500056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.500042 5991 scope.go:122] "RemoveContainer" containerID="820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55" Oct 30 20:15:04.500425 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:04.500406 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt_odh-ai-gateway-infra(5cae9d66-d044-4547-b344-feba5cd776f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" Oct 30 20:15:04.510029 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.510000 5991 scope.go:122] "RemoveContainer" containerID="7f536938f430b03b744ebf45623ac8803d7bc0b2518b02b46368d1ce3bcb0d28" Oct 30 20:15:04.524846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.524818 5991 scope.go:122] "RemoveContainer" containerID="d6a98c3d1fe1558d66b5e5c51311102caea9396fd88f0f7708c5b1363650ef94" Oct 30 20:15:04.689563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:04.689512 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" path="/var/lib/kubelet/pods/0123ae97-5cff-49f0-ab2b-c8261ef9bf7a/volumes" Oct 30 20:15:05.510716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:05.510681 5991 scope.go:122] "RemoveContainer" containerID="522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f" Oct 30 20:15:05.511162 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:05.511113 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx_odh-ai-gateway-infra(aad70126-a20e-4a9a-a96e-54765276567d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" podUID="aad70126-a20e-4a9a-a96e-54765276567d" Oct 30 20:15:05.512766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:05.512740 5991 scope.go:122] "RemoveContainer" containerID="820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55" Oct 30 20:15:05.513174 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:05.513150 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt_odh-ai-gateway-infra(5cae9d66-d044-4547-b344-feba5cd776f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" Oct 30 20:15:05.514621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:05.514600 5991 scope.go:122] "RemoveContainer" containerID="bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d" Oct 30 20:15:05.514939 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:05.514920 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-s5bjb_odh-ai-gateway-infra(fad16761-0452-4aa7-a2b1-118c4062f370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" Oct 30 20:15:06.099016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.098976 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:15:06.099223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.099202 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" podUID="cac1dad7-458c-438e-8b4c-8f910268d663" containerName="maas-api" containerID="cri-o://c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7" gracePeriod=30 Oct 30 20:15:06.434835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.429199 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:15:06.525445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.525411 5991 generic.go:356] "Generic (PLEG): container finished" podID="cac1dad7-458c-438e-8b4c-8f910268d663" containerID="c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7" exitCode=0 Oct 30 20:15:06.525887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.525465 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" Oct 30 20:15:06.525887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.525498 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" event={"ID":"cac1dad7-458c-438e-8b4c-8f910268d663","Type":"ContainerDied","Data":"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7"} Oct 30 20:15:06.525887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.525552 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd" event={"ID":"cac1dad7-458c-438e-8b4c-8f910268d663","Type":"ContainerDied","Data":"b09a7b2414aab1325ea08251a38f55cdd27a71353e0e5a507ea331f81ebad98f"} Oct 30 20:15:06.525887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.525575 5991 scope.go:122] "RemoveContainer" containerID="c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7" Oct 30 20:15:06.537123 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.537096 5991 scope.go:122] "RemoveContainer" containerID="c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7" Oct 30 20:15:06.537587 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:06.537553 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7\": container with ID starting with c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7 not found: ID does not exist" containerID="c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7" Oct 30 20:15:06.537690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.537600 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7"} err="failed to get container status \"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7\": rpc error: code = NotFound desc = could not find container \"c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7\": container with ID starting with c420cf6abe7fad47878a0d973fee410dce1ed72fe1605fc0684c828182a6d8f7 not found: ID does not exist" Oct 30 20:15:06.571518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.571481 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\") pod \"cac1dad7-458c-438e-8b4c-8f910268d663\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " Oct 30 20:15:06.571702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.571595 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\" (UniqueName: \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\") pod \"cac1dad7-458c-438e-8b4c-8f910268d663\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " Oct 30 20:15:06.571702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.571684 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\") pod \"cac1dad7-458c-438e-8b4c-8f910268d663\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " Oct 30 20:15:06.571880 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.571866 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") pod \"cac1dad7-458c-438e-8b4c-8f910268d663\" (UID: \"cac1dad7-458c-438e-8b4c-8f910268d663\") " Oct 30 20:15:06.572114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.572082 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca" pod "cac1dad7-458c-438e-8b4c-8f910268d663" (UID: "cac1dad7-458c-438e-8b4c-8f910268d663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:06.572851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.572810 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac1dad7-458c-438e-8b4c-8f910268d663-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:06.574666 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.574636 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls" pod "cac1dad7-458c-438e-8b4c-8f910268d663" (UID: "cac1dad7-458c-438e-8b4c-8f910268d663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:06.575060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.575025 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs" pod "cac1dad7-458c-438e-8b4c-8f910268d663" (UID: "cac1dad7-458c-438e-8b4c-8f910268d663"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:06.575879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.575854 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl" pod "cac1dad7-458c-438e-8b4c-8f910268d663" (UID: "cac1dad7-458c-438e-8b4c-8f910268d663"). InnerVolumeSpecName "kube-api-access-d5dnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:06.674933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.674845 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dnl\" (UniqueName: \"kubernetes.io/projected/cac1dad7-458c-438e-8b4c-8f910268d663-kube-api-access-d5dnl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:06.674933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.674878 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:06.674933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.674888 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac1dad7-458c-438e-8b4c-8f910268d663-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:06.866154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.866115 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:15:06.878564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:06.878527 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebe57b-56fcf9b6fb-kggxd"] Oct 30 20:15:08.681468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:08.681432 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac1dad7-458c-438e-8b4c-8f910268d663" path="/var/lib/kubelet/pods/cac1dad7-458c-438e-8b4c-8f910268d663/volumes" Oct 30 20:15:09.526971 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.526928 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5"] Oct 30 20:15:09.527795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527778 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" Oct 30 20:15:09.527874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527796 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" Oct 30 20:15:09.527874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527814 5991 cpu_manager.go:414] "containerMap: removing container" podUID="cac1dad7-458c-438e-8b4c-8f910268d663" containerName="maas-api" Oct 30 20:15:09.527874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527820 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac1dad7-458c-438e-8b4c-8f910268d663" containerName="maas-api" Oct 30 20:15:09.527973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527947 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0123ae97-5cff-49f0-ab2b-c8261ef9bf7a" containerName="istio-proxy" Oct 30 20:15:09.527973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.527959 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac1dad7-458c-438e-8b4c-8f910268d663" containerName="maas-api" Oct 30 20:15:09.533492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.533463 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.551063 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.551024 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5"] Oct 30 20:15:09.558096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.558065 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:09.636218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.636173 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.636509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.636480 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmc7\" (UniqueName: \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.636598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.636547 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.737504 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.737473 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.737907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.737678 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmc7\" (UniqueName: \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.737907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.737727 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.738209 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.738186 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.740233 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.740204 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.757511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.757469 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmc7\" (UniqueName: \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\") pod \"maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:09.853468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:09.853350 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:10.164499 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:15:10.164466 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3294f7f3_588f_4d18_a238_13137f6febe0.slice/crio-616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2 WatchSource:0}: Error finding container 616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2: Status 404 returned error can't find the container with id 616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2 Oct 30 20:15:10.165636 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:10.165605 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5"] Oct 30 20:15:10.576337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:10.576287 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" event={"ID":"3294f7f3-588f-4d18-a238-13137f6febe0","Type":"ContainerStarted","Data":"616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2"} Oct 30 20:15:11.588125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:11.588025 5991 generic.go:356] "Generic (PLEG): container finished" podID="3294f7f3-588f-4d18-a238-13137f6febe0" containerID="d184521506d5ad223f2617fba83dc92478b7f9f9f7e31c5b86a540c749b94e02" exitCode=0 Oct 30 20:15:11.588125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:11.588105 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" event={"ID":"3294f7f3-588f-4d18-a238-13137f6febe0","Type":"ContainerDied","Data":"d184521506d5ad223f2617fba83dc92478b7f9f9f7e31c5b86a540c749b94e02"} Oct 30 20:15:12.770134 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.770104 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:12.792042 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.792006 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\") pod \"3294f7f3-588f-4d18-a238-13137f6febe0\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " Oct 30 20:15:12.792042 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.792050 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\") pod \"3294f7f3-588f-4d18-a238-13137f6febe0\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " Oct 30 20:15:12.792239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.792080 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\" (UniqueName: \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\") pod \"3294f7f3-588f-4d18-a238-13137f6febe0\" (UID: \"3294f7f3-588f-4d18-a238-13137f6febe0\") " Oct 30 20:15:12.792549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.792521 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca" pod "3294f7f3-588f-4d18-a238-13137f6febe0" (UID: "3294f7f3-588f-4d18-a238-13137f6febe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:12.794398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.794345 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca" pod "3294f7f3-588f-4d18-a238-13137f6febe0" (UID: "3294f7f3-588f-4d18-a238-13137f6febe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:12.794527 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.794424 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7" pod "3294f7f3-588f-4d18-a238-13137f6febe0" (UID: "3294f7f3-588f-4d18-a238-13137f6febe0"). InnerVolumeSpecName "kube-api-access-7nmc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:12.893834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.893735 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:12.893834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.893775 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3294f7f3-588f-4d18-a238-13137f6febe0-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:12.893834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:12.893786 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nmc7\" (UniqueName: \"kubernetes.io/projected/3294f7f3-588f-4d18-a238-13137f6febe0-kube-api-access-7nmc7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:13.605551 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:13.605517 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" Oct 30 20:15:13.605897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:13.605553 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5" event={"ID":"3294f7f3-588f-4d18-a238-13137f6febe0","Type":"ContainerDied","Data":"616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2"} Oct 30 20:15:13.605897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:13.605614 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="616a67a7c1caa7b3319495fa9554657ae15b593f48a79374bd95531f7400bad2" Oct 30 20:15:14.122681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:14.122636 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:15:14.123761 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:14.123416 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" containerID="cri-o://a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8" gracePeriod=30 Oct 30 20:15:15.670988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:15.670939 5991 scope.go:122] "RemoveContainer" containerID="820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55" Oct 30 20:15:15.671577 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:15.671159 5991 scope.go:122] "RemoveContainer" containerID="bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d" Oct 30 20:15:15.671702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:15.671680 5991 scope.go:122] "RemoveContainer" containerID="522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f" Oct 30 20:15:16.634986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:16.634949 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerStarted","Data":"8cab32cc6173bf6988ab0c02be70a4812e0fc971b548888d70d792ccf1ebe67e"} Oct 30 20:15:16.637183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:16.637148 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerStarted","Data":"96333366c6788a2ce8805a22f2a55b24657b072a2614468f214490f5fa08efe8"} Oct 30 20:15:16.639138 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:16.639113 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerStarted","Data":"ce95c26dbd4e5de476d7f00253898ab30e66802a4cb2506f4bca9d21f03069dd"} Oct 30 20:15:17.648185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.648147 5991 generic.go:356] "Generic (PLEG): container finished" podID="aad70126-a20e-4a9a-a96e-54765276567d" containerID="ce95c26dbd4e5de476d7f00253898ab30e66802a4cb2506f4bca9d21f03069dd" exitCode=60 Oct 30 20:15:17.648737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.648225 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerDied","Data":"ce95c26dbd4e5de476d7f00253898ab30e66802a4cb2506f4bca9d21f03069dd"} Oct 30 20:15:17.648737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.648290 5991 scope.go:122] "RemoveContainer" containerID="522a537fec479da30057c5b2077dc9685e0e8b064d271c06bd6a4ebe1fdf277f" Oct 30 20:15:17.648842 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.648753 5991 scope.go:122] "RemoveContainer" containerID="ce95c26dbd4e5de476d7f00253898ab30e66802a4cb2506f4bca9d21f03069dd" Oct 30 20:15:17.649584 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:17.649179 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx_odh-ai-gateway-infra(aad70126-a20e-4a9a-a96e-54765276567d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" podUID="aad70126-a20e-4a9a-a96e-54765276567d" Oct 30 20:15:17.650883 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.650859 5991 generic.go:356] "Generic (PLEG): container finished" podID="5cae9d66-d044-4547-b344-feba5cd776f1" containerID="8cab32cc6173bf6988ab0c02be70a4812e0fc971b548888d70d792ccf1ebe67e" exitCode=60 Oct 30 20:15:17.650982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.650885 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerDied","Data":"8cab32cc6173bf6988ab0c02be70a4812e0fc971b548888d70d792ccf1ebe67e"} Oct 30 20:15:17.651409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.651344 5991 scope.go:122] "RemoveContainer" containerID="8cab32cc6173bf6988ab0c02be70a4812e0fc971b548888d70d792ccf1ebe67e" Oct 30 20:15:17.651746 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:17.651722 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt_odh-ai-gateway-infra(5cae9d66-d044-4547-b344-feba5cd776f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" Oct 30 20:15:17.653215 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.653194 5991 generic.go:356] "Generic (PLEG): container finished" podID="fad16761-0452-4aa7-a2b1-118c4062f370" containerID="96333366c6788a2ce8805a22f2a55b24657b072a2614468f214490f5fa08efe8" exitCode=60 Oct 30 20:15:17.653320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.653236 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerDied","Data":"96333366c6788a2ce8805a22f2a55b24657b072a2614468f214490f5fa08efe8"} Oct 30 20:15:17.653655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.653640 5991 scope.go:122] "RemoveContainer" containerID="96333366c6788a2ce8805a22f2a55b24657b072a2614468f214490f5fa08efe8" Oct 30 20:15:17.653975 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:17.653959 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-s5bjb_odh-ai-gateway-infra(fad16761-0452-4aa7-a2b1-118c4062f370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" Oct 30 20:15:17.663447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.663415 5991 scope.go:122] "RemoveContainer" containerID="820c910a3abb445c708b4c5df7c674c9d52ecd1470fc542a9e2ba3142b9e6c55" Oct 30 20:15:17.678817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.678792 5991 scope.go:122] "RemoveContainer" containerID="bb6906e643eaaf03e38865c27dca6a602500866fae0e158132dd369011e9681d" Oct 30 20:15:17.685822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.685790 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:17.723026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.722985 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:17.764584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:17.764536 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:18.869295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.868912 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:18.933843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.933810 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:18.979968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.979938 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:18.990970 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.990935 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\") pod \"aad70126-a20e-4a9a-a96e-54765276567d\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " Oct 30 20:15:18.991144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.991084 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\" (UniqueName: \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\") pod \"5cae9d66-d044-4547-b344-feba5cd776f1\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " Oct 30 20:15:18.991260 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.991244 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\") pod \"5cae9d66-d044-4547-b344-feba5cd776f1\" (UID: \"5cae9d66-d044-4547-b344-feba5cd776f1\") " Oct 30 20:15:18.991309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.991296 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\" (UniqueName: \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\") pod \"aad70126-a20e-4a9a-a96e-54765276567d\" (UID: \"aad70126-a20e-4a9a-a96e-54765276567d\") " Oct 30 20:15:18.991416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.991388 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca" pod "aad70126-a20e-4a9a-a96e-54765276567d" (UID: "aad70126-a20e-4a9a-a96e-54765276567d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:18.991760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.991731 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca" pod "5cae9d66-d044-4547-b344-feba5cd776f1" (UID: "5cae9d66-d044-4547-b344-feba5cd776f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:18.992197 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.992173 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad70126-a20e-4a9a-a96e-54765276567d-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.992296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.992203 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9d66-d044-4547-b344-feba5cd776f1-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.993622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.993589 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l" pod "5cae9d66-d044-4547-b344-feba5cd776f1" (UID: "5cae9d66-d044-4547-b344-feba5cd776f1"). InnerVolumeSpecName "kube-api-access-rth4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:18.994061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:18.994039 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz" pod "aad70126-a20e-4a9a-a96e-54765276567d" (UID: "aad70126-a20e-4a9a-a96e-54765276567d"). InnerVolumeSpecName "kube-api-access-bn4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.093837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.093800 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\" (UniqueName: \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\") pod \"fad16761-0452-4aa7-a2b1-118c4062f370\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " Oct 30 20:15:19.094054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.094039 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\") pod \"fad16761-0452-4aa7-a2b1-118c4062f370\" (UID: \"fad16761-0452-4aa7-a2b1-118c4062f370\") " Oct 30 20:15:19.094410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.094382 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca" pod "fad16761-0452-4aa7-a2b1-118c4062f370" (UID: "fad16761-0452-4aa7-a2b1-118c4062f370"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.094557 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.094542 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rth4l\" (UniqueName: \"kubernetes.io/projected/5cae9d66-d044-4547-b344-feba5cd776f1-kube-api-access-rth4l\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.094614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.094564 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4bz\" (UniqueName: \"kubernetes.io/projected/aad70126-a20e-4a9a-a96e-54765276567d-kube-api-access-bn4bz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.094614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.094579 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad16761-0452-4aa7-a2b1-118c4062f370-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.096097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.096069 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv" pod "fad16761-0452-4aa7-a2b1-118c4062f370" (UID: "fad16761-0452-4aa7-a2b1-118c4062f370"). InnerVolumeSpecName "kube-api-access-x7mpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.196023 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.195922 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7mpv\" (UniqueName: \"kubernetes.io/projected/fad16761-0452-4aa7-a2b1-118c4062f370-kube-api-access-x7mpv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.394253 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.394223 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:15:19.500105 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500072 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500115 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500218 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500260 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500328 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500451 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:19.500588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500469 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500705 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500602 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500713 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.500767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500602 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:19.500867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500641 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:19.500867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500802 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:19.500977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.500925 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\") pod \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\" (UID: \"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0\") " Oct 30 20:15:19.501224 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501186 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.501724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501704 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.501836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501727 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.501836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501741 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.501836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501753 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.501836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.501764 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.502879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.502857 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.503094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.503075 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:19.503252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.503233 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "kube-api-access-26w6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.503312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.503233 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo" pod "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" (UID: "f8271e7d-15c4-43ce-8b7a-04ed6db7afc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:19.602950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.602903 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26w6p\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-kube-api-access-26w6p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.602950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.602945 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.602950 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.602959 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.603194 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.602971 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.675772 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.675737 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" Oct 30 20:15:19.675980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.675770 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb" event={"ID":"fad16761-0452-4aa7-a2b1-118c4062f370","Type":"ContainerDied","Data":"3a6f898ad8ef096d577d3a60d8250038c61bbe83d0b3bb5f39dbb2d055e3d9b0"} Oct 30 20:15:19.675980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.675829 5991 scope.go:122] "RemoveContainer" containerID="96333366c6788a2ce8805a22f2a55b24657b072a2614468f214490f5fa08efe8" Oct 30 20:15:19.677547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.677527 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" Oct 30 20:15:19.677652 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.677524 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx" event={"ID":"aad70126-a20e-4a9a-a96e-54765276567d","Type":"ContainerDied","Data":"c52cc3b7406c63ba36f80f01eb4860fc5ff2ae007daa94d2400365ea65391d9f"} Oct 30 20:15:19.679162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.679133 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" Oct 30 20:15:19.679295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.679163 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt" event={"ID":"5cae9d66-d044-4547-b344-feba5cd776f1","Type":"ContainerDied","Data":"85c2996875d904f74c4a26f857d7b9c84315c9c6fd7cabae3b8a76103cd02456"} Oct 30 20:15:19.680716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.680689 5991 generic.go:356] "Generic (PLEG): container finished" podID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerID="a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8" exitCode=0 Oct 30 20:15:19.680846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.680729 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" event={"ID":"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0","Type":"ContainerDied","Data":"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8"} Oct 30 20:15:19.680846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.680738 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" Oct 30 20:15:19.680846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.680765 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7" event={"ID":"f8271e7d-15c4-43ce-8b7a-04ed6db7afc0","Type":"ContainerDied","Data":"bab573665d6765d93da83b911e9f637a528cb37c1cec715ae3975384c12b138f"} Oct 30 20:15:19.690116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.690093 5991 scope.go:122] "RemoveContainer" containerID="ce95c26dbd4e5de476d7f00253898ab30e66802a4cb2506f4bca9d21f03069dd" Oct 30 20:15:19.705052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.705026 5991 scope.go:122] "RemoveContainer" containerID="8cab32cc6173bf6988ab0c02be70a4812e0fc971b548888d70d792ccf1ebe67e" Oct 30 20:15:19.716609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.716583 5991 scope.go:122] "RemoveContainer" containerID="a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8" Oct 30 20:15:19.728248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.728224 5991 scope.go:122] "RemoveContainer" containerID="a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8" Oct 30 20:15:19.728637 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:19.728615 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8\": container with ID starting with a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8 not found: ID does not exist" containerID="a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8" Oct 30 20:15:19.728691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.728649 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8"} err="failed to get container status \"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8\": rpc error: code = NotFound desc = could not find container \"a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8\": container with ID starting with a597dbfe857408a441bd4530b39294a6486fe3d2b0d2e732cd39dfaa7e8c7ee8 not found: ID does not exist" Oct 30 20:15:19.777148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.777045 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:15:19.784502 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.784467 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-nhkz7"] Oct 30 20:15:19.846450 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.828907 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:19.847542 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.847432 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-s5bjb"] Oct 30 20:15:19.886651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.886617 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:19.895613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.895583 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a3426-29889855-rs8dt"] Oct 30 20:15:19.931091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.931046 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:19.947166 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:19.947105 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889855-m9fjx"] Oct 30 20:15:20.681231 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:20.681192 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" path="/var/lib/kubelet/pods/5cae9d66-d044-4547-b344-feba5cd776f1/volumes" Oct 30 20:15:20.681779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:20.681762 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad70126-a20e-4a9a-a96e-54765276567d" path="/var/lib/kubelet/pods/aad70126-a20e-4a9a-a96e-54765276567d/volumes" Oct 30 20:15:20.682243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:20.682230 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" path="/var/lib/kubelet/pods/f8271e7d-15c4-43ce-8b7a-04ed6db7afc0/volumes" Oct 30 20:15:20.683123 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:20.683099 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" path="/var/lib/kubelet/pods/fad16761-0452-4aa7-a2b1-118c4062f370/volumes" Oct 30 20:15:20.870813 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:20.870777 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:15:25.437109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:25.437067 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:15:27.764421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:27.764383 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:15:30.437742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:30.437679 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" containerID="cri-o://73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe" gracePeriod=25 Oct 30 20:15:33.138098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:33.138060 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:15:33.145773 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:33.145744 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:15:37.970293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:37.970253 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:40.292600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.292562 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:15:40.293416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.293345 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" podUID="3bdcd795-7d67-4491-bf73-1e739ffac88a" containerName="maas-api" containerID="cri-o://c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a" gracePeriod=30 Oct 30 20:15:40.603057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.603023 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:15:40.763841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.763797 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\") pod \"3bdcd795-7d67-4491-bf73-1e739ffac88a\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " Oct 30 20:15:40.764037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.763907 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\" (UniqueName: \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\") pod \"3bdcd795-7d67-4491-bf73-1e739ffac88a\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " Oct 30 20:15:40.764037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.764005 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\") pod \"3bdcd795-7d67-4491-bf73-1e739ffac88a\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " Oct 30 20:15:40.764245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.764219 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\") pod \"3bdcd795-7d67-4491-bf73-1e739ffac88a\" (UID: \"3bdcd795-7d67-4491-bf73-1e739ffac88a\") " Oct 30 20:15:40.765137 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.765106 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca" pod "3bdcd795-7d67-4491-bf73-1e739ffac88a" (UID: "3bdcd795-7d67-4491-bf73-1e739ffac88a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:40.766095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.766068 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdcd795-7d67-4491-bf73-1e739ffac88a-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:40.768343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.768300 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs" pod "3bdcd795-7d67-4491-bf73-1e739ffac88a" (UID: "3bdcd795-7d67-4491-bf73-1e739ffac88a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:40.773486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.773446 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls" pod "3bdcd795-7d67-4491-bf73-1e739ffac88a" (UID: "3bdcd795-7d67-4491-bf73-1e739ffac88a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:40.775932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.775877 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9" pod "3bdcd795-7d67-4491-bf73-1e739ffac88a" (UID: "3bdcd795-7d67-4491-bf73-1e739ffac88a"). InnerVolumeSpecName "kube-api-access-sk2w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:40.866331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.866235 5991 generic.go:356] "Generic (PLEG): container finished" podID="3bdcd795-7d67-4491-bf73-1e739ffac88a" containerID="c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a" exitCode=0 Oct 30 20:15:40.866758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.866734 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" event={"ID":"3bdcd795-7d67-4491-bf73-1e739ffac88a","Type":"ContainerDied","Data":"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a"} Oct 30 20:15:40.867039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.867020 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" event={"ID":"3bdcd795-7d67-4491-bf73-1e739ffac88a","Type":"ContainerDied","Data":"74c85de4c5c375d64142c03546e116aecab2ce2db43c0cf55df811f76dfaa9d5"} Oct 30 20:15:40.867180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.867167 5991 scope.go:122] "RemoveContainer" containerID="c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a" Oct 30 20:15:40.867774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.867752 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4" Oct 30 20:15:40.869382 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.869336 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:40.869485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.869395 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdcd795-7d67-4491-bf73-1e739ffac88a-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:40.869485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.869411 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk2w9\" (UniqueName: \"kubernetes.io/projected/3bdcd795-7d67-4491-bf73-1e739ffac88a-kube-api-access-sk2w9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:40.893675 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.893645 5991 scope.go:122] "RemoveContainer" containerID="c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a" Oct 30 20:15:40.894070 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:40.894047 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a\": container with ID starting with c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a not found: ID does not exist" containerID="c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a" Oct 30 20:15:40.894121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.894081 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a"} err="failed to get container status \"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a\": rpc error: code = NotFound desc = could not find container \"c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a\": container with ID starting with c470be27e546a58136508b5ea027d12bb15e3bad095384b9296ff6787f43984a not found: ID does not exist" Oct 30 20:15:40.920584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.920552 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:15:40.932719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:40.932679 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d72c-8498b4dd5b-2scb4"] Oct 30 20:15:42.680981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:42.680945 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bdcd795-7d67-4491-bf73-1e739ffac88a" path="/var/lib/kubelet/pods/3bdcd795-7d67-4491-bf73-1e739ffac88a/volumes" Oct 30 20:15:47.970971 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:47.970928 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:50.753094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.753057 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5"] Oct 30 20:15:50.754462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.754443 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:15:50.764013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.763978 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cd2ad-kt5l5"] Oct 30 20:15:50.827719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.827673 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\") pod \"0d8a9ae0-85ce-44a4-b298-40198f94b251\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " Oct 30 20:15:50.827910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.827874 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\" (UniqueName: \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\") pod \"0d8a9ae0-85ce-44a4-b298-40198f94b251\" (UID: \"0d8a9ae0-85ce-44a4-b298-40198f94b251\") " Oct 30 20:15:50.828207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.828178 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume" pod "0d8a9ae0-85ce-44a4-b298-40198f94b251" (UID: "0d8a9ae0-85ce-44a4-b298-40198f94b251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:50.828401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.828379 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8a9ae0-85ce-44a4-b298-40198f94b251-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.830109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.830052 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk" pod "0d8a9ae0-85ce-44a4-b298-40198f94b251" (UID: "0d8a9ae0-85ce-44a4-b298-40198f94b251"). InnerVolumeSpecName "kube-api-access-8lpwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:50.872816 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:50.872772 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:15:50.930692 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.930654 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpwk\" (UniqueName: \"kubernetes.io/projected/0d8a9ae0-85ce-44a4-b298-40198f94b251-kube-api-access-8lpwk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.955884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.955845 5991 generic.go:356] "Generic (PLEG): container finished" podID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerID="73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe" exitCode=0 Oct 30 20:15:50.956102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.955899 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" Oct 30 20:15:50.956102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.955927 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" event={"ID":"0d8a9ae0-85ce-44a4-b298-40198f94b251","Type":"ContainerDied","Data":"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe"} Oct 30 20:15:50.956102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.955989 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs" event={"ID":"0d8a9ae0-85ce-44a4-b298-40198f94b251","Type":"ContainerDied","Data":"2f91f7869cfd2964f4d5a6ba4e764dd36677bfb8a441603168973651fe8ce9cf"} Oct 30 20:15:50.956102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.956017 5991 scope.go:122] "RemoveContainer" containerID="73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe" Oct 30 20:15:50.967992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.967965 5991 scope.go:122] "RemoveContainer" containerID="73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe" Oct 30 20:15:50.968317 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:15:50.968297 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe\": container with ID starting with 73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe not found: ID does not exist" containerID="73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe" Oct 30 20:15:50.968430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:50.968326 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe"} err="failed to get container status \"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe\": rpc error: code = NotFound desc = could not find container \"73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe\": container with ID starting with 73237333e82d2e5d147a75e831be32f274b1600c19c5ffce8aac0a6244ab9afe not found: ID does not exist" Oct 30 20:15:51.008953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:51.008919 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:15:51.013656 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:51.013618 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a31d3-69b6b6d6f8-s2jbs"] Oct 30 20:15:52.272296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.272251 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:15:52.273043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273025 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3bdcd795-7d67-4491-bf73-1e739ffac88a" containerName="maas-api" Oct 30 20:15:52.273085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273046 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bdcd795-7d67-4491-bf73-1e739ffac88a" containerName="maas-api" Oct 30 20:15:52.273085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273064 5991 cpu_manager.go:414] "containerMap: removing container" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273069 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273079 5991 cpu_manager.go:414] "containerMap: removing container" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273084 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273101 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273106 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273114 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273118 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273128 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273133 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273138 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3294f7f3-588f-4d18-a238-13137f6febe0" containerName="revoke-keys" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273142 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3294f7f3-588f-4d18-a238-13137f6febe0" containerName="revoke-keys" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273153 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273158 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273167 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273171 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273178 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273182 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273206 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" Oct 30 20:15:52.273248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273210 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273336 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273349 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273376 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273388 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8271e7d-15c4-43ce-8b7a-04ed6db7afc0" containerName="istio-proxy" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273413 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273426 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bdcd795-7d67-4491-bf73-1e739ffac88a" containerName="maas-api" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273437 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273446 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273458 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" containerName="payload-processing" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273469 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:15:52.273832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.273477 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3294f7f3-588f-4d18-a238-13137f6febe0" containerName="revoke-keys" Oct 30 20:15:52.279629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.279603 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.294575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.294541 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a165e5-openshift-default-dockercfg-wc9pt\"" Oct 30 20:15:52.312245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.312204 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:15:52.352545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352506 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75g2\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352567 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352672 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352808 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352840 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352875 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.352941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.352898 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.353021 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.353003 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.353094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.353082 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454222 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454309 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454421 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75g2\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454465 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454504 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454602 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454630 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454684 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454721 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.454895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.454791 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.455249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.455221 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.455430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.455310 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.455517 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.455438 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.456038 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.456011 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.457245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.457222 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.457547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.457524 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.474480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.474445 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.474634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.474593 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75g2\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.596066 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.595970 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:52.686969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.686930 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8a9ae0-85ce-44a4-b298-40198f94b251" path="/var/lib/kubelet/pods/0d8a9ae0-85ce-44a4-b298-40198f94b251/volumes" Oct 30 20:15:52.687612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.687579 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3294f7f3-588f-4d18-a238-13137f6febe0" path="/var/lib/kubelet/pods/3294f7f3-588f-4d18-a238-13137f6febe0/volumes" Oct 30 20:15:52.919959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.919928 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:15:52.921091 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:15:52.921062 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e43405_b235_4f7d_9589_555f3d20bee4.slice/crio-5f0dfdbd710b41c1443a15a6767725f59fe4c8b843f0fdc92d8630623ff10163 WatchSource:0}: Error finding container 5f0dfdbd710b41c1443a15a6767725f59fe4c8b843f0fdc92d8630623ff10163: Status 404 returned error can't find the container with id 5f0dfdbd710b41c1443a15a6767725f59fe4c8b843f0fdc92d8630623ff10163 Oct 30 20:15:52.924677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.924623 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:52.924840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.924719 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:52.924840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.924773 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:52.974854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:52.974820 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" event={"ID":"55e43405-b235-4f7d-9589-555f3d20bee4","Type":"ContainerStarted","Data":"5f0dfdbd710b41c1443a15a6767725f59fe4c8b843f0fdc92d8630623ff10163"} Oct 30 20:15:53.995531 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:53.995491 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" event={"ID":"55e43405-b235-4f7d-9589-555f3d20bee4","Type":"ContainerStarted","Data":"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f"} Oct 30 20:15:54.062452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:54.062384 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podStartSLOduration=2.062341833 podStartE2EDuration="2.062341833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:54.049904967 +0000 UTC m=+2421.499397692" watchObservedRunningTime="2026-10-30 20:15:54.062341833 +0000 UTC m=+2421.511834560" Oct 30 20:15:54.596238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:54.596192 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:54.598775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:54.598744 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:15:54.598915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:54.598802 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 20:15:55.596553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:55.596512 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:15:55.597113 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:55.596578 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 20:15:56.597068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:56.597022 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:15:56.597588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:56.597108 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 20:15:57.600912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:57.600875 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:15:57.601572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:57.601536 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 20:15:58.215726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.215669 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:15:58.216599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.216543 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" containerID="cri-o://b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd" gracePeriod=30 Oct 30 20:15:58.221586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.221551 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:15:58.221839 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.221810 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" containerID="cri-o://92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3" gracePeriod=30 Oct 30 20:15:58.599398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.598678 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:15:58.599398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:58.598784 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 20:15:59.739143 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:59.739110 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:59.739758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:59.739635 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:15:59.740549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:15:59.740524 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:16:03.534463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.534430 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:16:03.580410 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.580350 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:16:03.595560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595521 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.595803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595594 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.595803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595622 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.595803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595692 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.595803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595745 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.595803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595804 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.596109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595844 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.596109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595863 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.596109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.595959 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.596109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596065 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.596109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596093 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.596514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596136 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.596514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596235 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.596514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596275 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.596514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596349 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.596514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596431 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\") pod \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\" (UID: \"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda\") " Oct 30 20:16:03.596765 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596645 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.596879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.596808 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.597193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.597168 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.597193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.597199 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.597430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.597214 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.597430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.597229 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.597590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.597476 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.598449 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.598266 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.598732 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.598686 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:03.598951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.598917 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "kube-api-access-zcgj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.599116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.599085 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.601541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.601514 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:03.602142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.602103 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "kube-api-access-k99c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.602142 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.602128 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.602288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.602138 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.602288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.602230 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy" pod "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" (UID: "b99e3810-2c66-4fd5-bbcf-c3785d8b7bda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.698441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698395 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.698648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698465 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.698648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698539 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.698648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698645 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\") pod \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\" (UID: \"dc85d5fa-ec65-4a88-84a3-fa10887840fd\") " Oct 30 20:16:03.698895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698834 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:03.699015 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.698913 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.699249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699232 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699251 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699260 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699268 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699275 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699283 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699290 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699299 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99c7\" (UniqueName: \"kubernetes.io/projected/dc85d5fa-ec65-4a88-84a3-fa10887840fd-kube-api-access-k99c7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699308 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699318 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699326 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcgj8\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-kube-api-access-zcgj8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.699546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.699334 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.700673 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.700650 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:03.700810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.700792 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy" pod "dc85d5fa-ec65-4a88-84a3-fa10887840fd" (UID: "dc85d5fa-ec65-4a88-84a3-fa10887840fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.801529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.801442 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:03.801529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:03.801478 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc85d5fa-ec65-4a88-84a3-fa10887840fd-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:16:04.076262 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.076174 5991 generic.go:356] "Generic (PLEG): container finished" podID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerID="92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3" exitCode=0 Oct 30 20:16:04.076262 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.076229 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" Oct 30 20:16:04.076545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.076260 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" event={"ID":"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda","Type":"ContainerDied","Data":"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3"} Oct 30 20:16:04.076545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.076318 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6" event={"ID":"b99e3810-2c66-4fd5-bbcf-c3785d8b7bda","Type":"ContainerDied","Data":"3f921aac48d9e61b734feb2dd5f5c2a29d6c35ded8e3cb4428ce8489b442589c"} Oct 30 20:16:04.076545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.076350 5991 scope.go:122] "RemoveContainer" containerID="92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3" Oct 30 20:16:04.078222 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.078193 5991 generic.go:356] "Generic (PLEG): container finished" podID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerID="b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd" exitCode=0 Oct 30 20:16:04.078415 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.078229 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" event={"ID":"dc85d5fa-ec65-4a88-84a3-fa10887840fd","Type":"ContainerDied","Data":"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd"} Oct 30 20:16:04.078415 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.078264 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" Oct 30 20:16:04.078415 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.078273 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn" event={"ID":"dc85d5fa-ec65-4a88-84a3-fa10887840fd","Type":"ContainerDied","Data":"00905c7769233a04372dd55526bf581e982da5bb97049fed1163d2b60c9deceb"} Oct 30 20:16:04.088571 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.088504 5991 scope.go:122] "RemoveContainer" containerID="92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3" Oct 30 20:16:04.088853 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:16:04.088832 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3\": container with ID starting with 92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3 not found: ID does not exist" containerID="92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3" Oct 30 20:16:04.088942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.088861 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3"} err="failed to get container status \"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3\": rpc error: code = NotFound desc = could not find container \"92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3\": container with ID starting with 92b8506883c4f93023ddf6fc8cdf47c85f409d21ebf7c4f820264a6e3b567af3 not found: ID does not exist" Oct 30 20:16:04.088942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.088883 5991 scope.go:122] "RemoveContainer" containerID="b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd" Oct 30 20:16:04.102906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.102877 5991 scope.go:122] "RemoveContainer" containerID="b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd" Oct 30 20:16:04.103337 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:16:04.103312 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd\": container with ID starting with b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd not found: ID does not exist" containerID="b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd" Oct 30 20:16:04.103466 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.103348 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd"} err="failed to get container status \"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd\": rpc error: code = NotFound desc = could not find container \"b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd\": container with ID starting with b3f74deb9af438528307bc5268ccbf8e3bb0ec5c97016e759c8a7310d3d6cfdd not found: ID does not exist" Oct 30 20:16:04.130152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.130116 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:16:04.142832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.142792 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-x8tj6"] Oct 30 20:16:04.176882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.176846 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:16:04.181256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.181216 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-qvngn"] Oct 30 20:16:04.681040 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.681004 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" path="/var/lib/kubelet/pods/b99e3810-2c66-4fd5-bbcf-c3785d8b7bda/volumes" Oct 30 20:16:04.681646 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.681629 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" path="/var/lib/kubelet/pods/dc85d5fa-ec65-4a88-84a3-fa10887840fd/volumes" Oct 30 20:16:04.985087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985050 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:16:04.985806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985777 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:16:04.985806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985809 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985846 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985853 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985880 5991 cpu_manager.go:414] "containerMap: removing container" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985885 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985895 5991 cpu_manager.go:414] "containerMap: removing container" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:16:04.985916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.985899 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad70126-a20e-4a9a-a96e-54765276567d" containerName="cleanup" Oct 30 20:16:04.986125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.986019 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad16761-0452-4aa7-a2b1-118c4062f370" containerName="cleanup" Oct 30 20:16:04.986125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.986031 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99e3810-2c66-4fd5-bbcf-c3785d8b7bda" containerName="istio-proxy" Oct 30 20:16:04.986125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.986039 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae9d66-d044-4547-b344-feba5cd776f1" containerName="cleanup" Oct 30 20:16:04.986125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.986049 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc85d5fa-ec65-4a88-84a3-fa10887840fd" containerName="istio-proxy" Oct 30 20:16:04.991216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:04.991188 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.007052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.007016 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-77ac5a\"" Oct 30 20:16:05.007228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.007201 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-77ac5a-dockercfg-f2njz\"" Oct 30 20:16:05.024240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.024200 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:16:05.051815 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.051584 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:16:05.123396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.120645 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrhf\" (UniqueName: \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.123396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.120790 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.150837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.135458 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:16:05.150837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.135591 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.221845 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.221809 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.222060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.221926 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6bz\" (UniqueName: \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.222131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.222088 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrhf\" (UniqueName: \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.222809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.222140 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.223173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.223127 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.242170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.242074 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrhf\" (UniqueName: \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\") pod \"payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.307151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.307089 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:05.325215 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.325171 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.326052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.325455 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6bz\" (UniqueName: \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.326326 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.326295 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.346887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.346838 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6bz\" (UniqueName: \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\") pod \"payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.460774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.460734 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:05.625894 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:16:05.625862 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02f4a3f_4d82_4688_8b80_83b7d2332378.slice/crio-8b18fa9d55d17b59d2ce317b7b68e854814d995643ef9b2cba510e4ccc99138b WatchSource:0}: Error finding container 8b18fa9d55d17b59d2ce317b7b68e854814d995643ef9b2cba510e4ccc99138b: Status 404 returned error can't find the container with id 8b18fa9d55d17b59d2ce317b7b68e854814d995643ef9b2cba510e4ccc99138b Oct 30 20:16:05.626597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.626565 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:16:05.776706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:05.776675 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:16:05.777118 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:16:05.777099 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901efb89_7a25_40ea_9fb5_8d3f95286f72.slice/crio-193964152ac7d1e8c10a9c65af3e20ae3c3181f6688ebb9614dc3bb864d618aa WatchSource:0}: Error finding container 193964152ac7d1e8c10a9c65af3e20ae3c3181f6688ebb9614dc3bb864d618aa: Status 404 returned error can't find the container with id 193964152ac7d1e8c10a9c65af3e20ae3c3181f6688ebb9614dc3bb864d618aa Oct 30 20:16:06.119708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.119594 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" event={"ID":"f02f4a3f-4d82-4688-8b80-83b7d2332378","Type":"ContainerStarted","Data":"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb"} Oct 30 20:16:06.119708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.119669 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" event={"ID":"f02f4a3f-4d82-4688-8b80-83b7d2332378","Type":"ContainerStarted","Data":"8b18fa9d55d17b59d2ce317b7b68e854814d995643ef9b2cba510e4ccc99138b"} Oct 30 20:16:06.119708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.119704 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:06.121420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.121354 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" event={"ID":"901efb89-7a25-40ea-9fb5-8d3f95286f72","Type":"ContainerStarted","Data":"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86"} Oct 30 20:16:06.121567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.121448 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" event={"ID":"901efb89-7a25-40ea-9fb5-8d3f95286f72","Type":"ContainerStarted","Data":"193964152ac7d1e8c10a9c65af3e20ae3c3181f6688ebb9614dc3bb864d618aa"} Oct 30 20:16:06.121567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.121490 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:06.162318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.162262 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" podStartSLOduration=2.162245761 podStartE2EDuration="2.162245761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:06.160586926 +0000 UTC m=+2433.610079653" watchObservedRunningTime="2026-10-30 20:16:06.162245761 +0000 UTC m=+2433.611738485" Oct 30 20:16:06.201684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:06.201624 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" podStartSLOduration=1.201608322 podStartE2EDuration="1.201608322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:06.199025142 +0000 UTC m=+2433.648517878" watchObservedRunningTime="2026-10-30 20:16:06.201608322 +0000 UTC m=+2433.651101046" Oct 30 20:16:14.061481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.061356 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:16:14.066612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.066579 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.085844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.085808 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:16:14.146386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.132813 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5314c2-dockercfg-zh2qq\"" Oct 30 20:16:14.146386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.132922 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5314c2\"" Oct 30 20:16:14.195749 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.195697 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.196243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.196216 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snv9z\" (UniqueName: \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.299060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.299009 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.299280 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.299186 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snv9z\" (UniqueName: \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.299956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.299929 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.319944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.319861 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snv9z\" (UniqueName: \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\") pod \"payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.388721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.388676 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:14.693322 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:14.693289 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:16:14.693471 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:16:14.693287 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c78352_08a5_4222_a2e0_5849f9ae8fcd.slice/crio-595dcd97cb16ffda610a2b14850864c7350d2aa3e92c3134d0dd3d4e91633525 WatchSource:0}: Error finding container 595dcd97cb16ffda610a2b14850864c7350d2aa3e92c3134d0dd3d4e91633525: Status 404 returned error can't find the container with id 595dcd97cb16ffda610a2b14850864c7350d2aa3e92c3134d0dd3d4e91633525 Oct 30 20:16:15.238608 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:15.238564 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" event={"ID":"13c78352-08a5-4222-a2e0-5849f9ae8fcd","Type":"ContainerStarted","Data":"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422"} Oct 30 20:16:15.239060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:15.238638 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" event={"ID":"13c78352-08a5-4222-a2e0-5849f9ae8fcd","Type":"ContainerStarted","Data":"595dcd97cb16ffda610a2b14850864c7350d2aa3e92c3134d0dd3d4e91633525"} Oct 30 20:16:15.239060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:15.238782 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:15.280182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:15.280119 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" podStartSLOduration=1.28010319 podStartE2EDuration="1.28010319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:15.277090052 +0000 UTC m=+2442.726582777" watchObservedRunningTime="2026-10-30 20:16:15.28010319 +0000 UTC m=+2442.729595915" Oct 30 20:16:17.133748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:17.133719 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:16:17.134147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:17.133789 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:16:20.870953 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:16:20.870917 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:16:23.521912 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:23.521870 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:16:26.249982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:26.249951 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:16:39.158684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.158646 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:16:39.165549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.165512 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.186052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.186011 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:16:39.191952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.191920 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-77ac5a-serving-cert\"" Oct 30 20:16:39.295529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.295489 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsz55\" (UniqueName: \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.295754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.295655 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.295754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.295715 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.295938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.295921 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.397666 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.397625 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.397904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.397775 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsz55\" (UniqueName: \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.397904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.397854 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.397904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.397898 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.398860 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.398828 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.400947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.400918 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.402124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.402097 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.418595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.418517 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsz55\" (UniqueName: \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\") pod \"maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.492475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.492433 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:39.803935 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:16:39.803906 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33cfc0d_ddbf_40df_bb30_e636a9a3aee2.slice/crio-c709f7c03e6d618a9642b0d6965c156855ea0d7164dc9c0ce3cfec8fd826c10b WatchSource:0}: Error finding container c709f7c03e6d618a9642b0d6965c156855ea0d7164dc9c0ce3cfec8fd826c10b: Status 404 returned error can't find the container with id c709f7c03e6d618a9642b0d6965c156855ea0d7164dc9c0ce3cfec8fd826c10b Oct 30 20:16:39.804720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.804687 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:16:39.805821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:39.805800 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:16:40.447274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:40.447221 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" event={"ID":"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2","Type":"ContainerStarted","Data":"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af"} Oct 30 20:16:40.447745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:40.447298 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" event={"ID":"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2","Type":"ContainerStarted","Data":"c709f7c03e6d618a9642b0d6965c156855ea0d7164dc9c0ce3cfec8fd826c10b"} Oct 30 20:16:40.447745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:40.447331 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:40.492113 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:40.492048 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" podStartSLOduration=1.049206095 podStartE2EDuration="1.49203236s" totalImagesPullingTime="442.826265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:40.490601953 +0000 UTC m=+2467.940094679" watchObservedRunningTime="2026-10-30 20:16:40.49203236 +0000 UTC m=+2467.941525084" Oct 30 20:16:46.461918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:16:46.461878 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:16:50.871109 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:16:50.871065 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:17:11.281292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:11.281251 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:17:16.282008 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:16.281955 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" containerID="cri-o://b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872" gracePeriod=25 Oct 30 20:17:20.869963 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:17:20.869917 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:17:21.069539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:21.069505 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:17:22.291739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:22.291700 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:32.291820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:32.291780 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:34.330634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:34.330598 5991 scope.go:122] "RemoveContainer" containerID="bb496389bc3c225ca24c9d75bca883d90f7ca399d11a4cf1640cbbe3d4831d86" Oct 30 20:17:36.092413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.092355 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" podUID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" containerName="main" containerID="cri-o://874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3" gracePeriod=45 Oct 30 20:17:36.427535 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.427498 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:17:36.568164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.568133 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:17:36.593952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.593919 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.593965 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594010 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\" (UniqueName: \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594085 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594141 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594195 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\") pod \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\" (UID: \"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2\") " Oct 30 20:17:36.594435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594242 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:36.594633 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594611 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:36.594750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.594712 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:36.595618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.595070 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.595618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.595094 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.595618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.595107 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.596805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.596780 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:36.596897 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.596882 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:36.597232 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.597210 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5" pod "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" (UID: "1b0b0519-3d28-4ec9-99c2-e2cebc3963c2"). InnerVolumeSpecName "kube-api-access-k9dm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:36.698282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.698250 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\" (UniqueName: \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\") pod \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " Oct 30 20:17:36.698727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.698698 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\") pod \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\" (UID: \"6f4b33c1-a470-485d-b2da-b818ae2f9ca5\") " Oct 30 20:17:36.699452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.699419 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume" pod "6f4b33c1-a470-485d-b2da-b818ae2f9ca5" (UID: "6f4b33c1-a470-485d-b2da-b818ae2f9ca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:36.699661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.699641 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.699745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.699666 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dm5\" (UniqueName: \"kubernetes.io/projected/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-kube-api-access-k9dm5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.699745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.699683 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.699745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.699696 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.701076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.700983 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp" pod "6f4b33c1-a470-485d-b2da-b818ae2f9ca5" (UID: "6f4b33c1-a470-485d-b2da-b818ae2f9ca5"). InnerVolumeSpecName "kube-api-access-4wmbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:36.801247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.801205 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wmbp\" (UniqueName: \"kubernetes.io/projected/6f4b33c1-a470-485d-b2da-b818ae2f9ca5-kube-api-access-4wmbp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.977593 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.977553 5991 generic.go:356] "Generic (PLEG): container finished" podID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerID="b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872" exitCode=0 Oct 30 20:17:36.977817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.977635 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" event={"ID":"6f4b33c1-a470-485d-b2da-b818ae2f9ca5","Type":"ContainerDied","Data":"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872"} Oct 30 20:17:36.977817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.977646 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" Oct 30 20:17:36.977817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.977695 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl" event={"ID":"6f4b33c1-a470-485d-b2da-b818ae2f9ca5","Type":"ContainerDied","Data":"e603e203b777d28bc82e5121812222f6bac1df4ab13cd37602ab8c8e7f07440b"} Oct 30 20:17:36.977817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.977723 5991 scope.go:122] "RemoveContainer" containerID="b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872" Oct 30 20:17:36.979561 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.979531 5991 generic.go:356] "Generic (PLEG): container finished" podID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" containerID="874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3" exitCode=0 Oct 30 20:17:36.979702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.979578 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" event={"ID":"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2","Type":"ContainerDied","Data":"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3"} Oct 30 20:17:36.979702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.979595 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" Oct 30 20:17:36.979702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.979607 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn" event={"ID":"1b0b0519-3d28-4ec9-99c2-e2cebc3963c2","Type":"ContainerDied","Data":"e564fe4f35c31d722a61e82e60cb5a7656687b0a193cbb86ea02dc487949cf67"} Oct 30 20:17:36.989579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.989555 5991 scope.go:122] "RemoveContainer" containerID="b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872" Oct 30 20:17:36.989926 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:17:36.989902 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872\": container with ID starting with b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872 not found: ID does not exist" containerID="b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872" Oct 30 20:17:36.990013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.989936 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872"} err="failed to get container status \"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872\": rpc error: code = NotFound desc = could not find container \"b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872\": container with ID starting with b6e81adbb705c99cc512c712e9e6dc87ea54de4f47aed464b3678fee12afd872 not found: ID does not exist" Oct 30 20:17:36.990013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:36.989956 5991 scope.go:122] "RemoveContainer" containerID="874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3" Oct 30 20:17:37.000401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.000373 5991 scope.go:122] "RemoveContainer" containerID="874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3" Oct 30 20:17:37.000724 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:17:37.000706 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3\": container with ID starting with 874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3 not found: ID does not exist" containerID="874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3" Oct 30 20:17:37.000795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.000734 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3"} err="failed to get container status \"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3\": rpc error: code = NotFound desc = could not find container \"874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3\": container with ID starting with 874712b7d4ba5b81c1a8f700946430e42b10efbc2aaf9bc19164ba16c6ad7ea3 not found: ID does not exist" Oct 30 20:17:37.030194 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.030159 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:17:37.033451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.033426 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8fc40-7b99896bc6-7gcjl"] Oct 30 20:17:37.066321 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.066286 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:17:37.071151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:37.071122 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-21c4ab/auth-test-model-21c4ab-kserve-549cffdcb5-c8wjn"] Oct 30 20:17:38.683838 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:38.683801 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" path="/var/lib/kubelet/pods/1b0b0519-3d28-4ec9-99c2-e2cebc3963c2/volumes" Oct 30 20:17:38.684315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:38.684298 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" path="/var/lib/kubelet/pods/6f4b33c1-a470-485d-b2da-b818ae2f9ca5/volumes" Oct 30 20:17:42.983956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:42.982902 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:17:42.983956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:42.983201 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" podUID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" containerName="maas-api" containerID="cri-o://1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73" gracePeriod=30 Oct 30 20:17:43.285229 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.285200 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:17:43.292589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.292553 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\" (UniqueName: \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\") pod \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " Oct 30 20:17:43.292789 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.292635 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\") pod \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " Oct 30 20:17:43.292789 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.292702 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\") pod \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " Oct 30 20:17:43.292926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.292825 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\") pod \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\" (UID: \"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c\") " Oct 30 20:17:43.293189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.293162 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca" pod "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" (UID: "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:43.293860 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.293840 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:43.294896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.294866 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs" pod "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" (UID: "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:43.295019 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.295005 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx" pod "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" (UID: "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c"). InnerVolumeSpecName "kube-api-access-jl2vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:43.295085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.295070 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls" pod "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" (UID: "79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:43.395568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.395529 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2vx\" (UniqueName: \"kubernetes.io/projected/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-kube-api-access-jl2vx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:43.395568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.395568 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:43.395784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:43.395583 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:17:44.043779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.043745 5991 generic.go:356] "Generic (PLEG): container finished" podID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" containerID="1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73" exitCode=0 Oct 30 20:17:44.044235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.043785 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" event={"ID":"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c","Type":"ContainerDied","Data":"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73"} Oct 30 20:17:44.044235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.043815 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" Oct 30 20:17:44.044235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.043841 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc" event={"ID":"79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c","Type":"ContainerDied","Data":"40436eead5c2adf603acdeb3ea5d9251b1f30b4c359d7ce9b261764c78b26cc4"} Oct 30 20:17:44.044235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.043863 5991 scope.go:122] "RemoveContainer" containerID="1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73" Oct 30 20:17:44.062663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.062636 5991 scope.go:122] "RemoveContainer" containerID="1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73" Oct 30 20:17:44.062970 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:17:44.062952 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73\": container with ID starting with 1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73 not found: ID does not exist" containerID="1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73" Oct 30 20:17:44.063014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.062980 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73"} err="failed to get container status \"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73\": rpc error: code = NotFound desc = could not find container \"1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73\": container with ID starting with 1b128a16468c4ec076b64262aa1502dfd1da1401755f292114d2f416f8574a73 not found: ID does not exist" Oct 30 20:17:44.095915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.095880 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:17:44.108324 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.108291 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8fc40-d4d764db4-h8hpc"] Oct 30 20:17:44.680740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:44.680703 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" path="/var/lib/kubelet/pods/79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c/volumes" Oct 30 20:17:46.265645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.265607 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:17:46.266464 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266447 5991 cpu_manager.go:414] "containerMap: removing container" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" Oct 30 20:17:46.266520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266465 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" Oct 30 20:17:46.266520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266486 5991 cpu_manager.go:414] "containerMap: removing container" podUID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" containerName="main" Oct 30 20:17:46.266520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266491 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" containerName="main" Oct 30 20:17:46.266520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266509 5991 cpu_manager.go:414] "containerMap: removing container" podUID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" containerName="maas-api" Oct 30 20:17:46.266520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266515 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" containerName="maas-api" Oct 30 20:17:46.266682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266633 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4b33c1-a470-485d-b2da-b818ae2f9ca5" containerName="payload-processing" Oct 30 20:17:46.266682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266645 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0b0519-3d28-4ec9-99c2-e2cebc3963c2" containerName="main" Oct 30 20:17:46.266682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.266654 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f571b1-6ea2-47e2-9b10-ba0fb05dcf5c" containerName="maas-api" Oct 30 20:17:46.271755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.271737 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.293095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.293052 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-5c875d\"" Oct 30 20:17:46.293335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.293267 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-5c875d-dockercfg-svbzs\"" Oct 30 20:17:46.310204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.310164 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:17:46.345745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.345711 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2x2z\" (UniqueName: \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.346258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.346229 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.371401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.370540 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:17:46.386386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.376904 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.410771 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.410710 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:17:46.448525 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.448487 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nts2\" (UniqueName: \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.448718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.448536 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2x2z\" (UniqueName: \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.448718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.448655 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.449011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.448986 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.454381 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.449905 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.469425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.469343 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2x2z\" (UniqueName: \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\") pod \"payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.550396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.550297 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nts2\" (UniqueName: \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.550396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.550348 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.551085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.551061 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.570190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.570148 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nts2\" (UniqueName: \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\") pod \"payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.587400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.587343 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:46.716480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.716443 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:46.898099 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:17:46.898067 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2253ed_47bd_4dfa_8340_6974b1130605.slice/crio-3ba3db9c1d3e3e590ee3b3e28fe4942757e7c1fd13b9f623b43fcde04f1b9272 WatchSource:0}: Error finding container 3ba3db9c1d3e3e590ee3b3e28fe4942757e7c1fd13b9f623b43fcde04f1b9272: Status 404 returned error can't find the container with id 3ba3db9c1d3e3e590ee3b3e28fe4942757e7c1fd13b9f623b43fcde04f1b9272 Oct 30 20:17:46.898289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:46.898267 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:17:47.034573 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:17:47.034533 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb5424_e691_4875_b046_f56eeffe32d4.slice/crio-c3fcb852700d4fb772f636fa3d636b33e626f9a435d4a3f0479785a86a027ec3 WatchSource:0}: Error finding container c3fcb852700d4fb772f636fa3d636b33e626f9a435d4a3f0479785a86a027ec3: Status 404 returned error can't find the container with id c3fcb852700d4fb772f636fa3d636b33e626f9a435d4a3f0479785a86a027ec3 Oct 30 20:17:47.034925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.034901 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:17:47.078355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.078320 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" event={"ID":"6d2253ed-47bd-4dfa-8340-6974b1130605","Type":"ContainerStarted","Data":"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915"} Oct 30 20:17:47.078355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.078393 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" event={"ID":"6d2253ed-47bd-4dfa-8340-6974b1130605","Type":"ContainerStarted","Data":"3ba3db9c1d3e3e590ee3b3e28fe4942757e7c1fd13b9f623b43fcde04f1b9272"} Oct 30 20:17:47.078600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.078419 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:47.080100 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.080073 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" event={"ID":"18bb5424-e691-4875-b046-f56eeffe32d4","Type":"ContainerStarted","Data":"c3fcb852700d4fb772f636fa3d636b33e626f9a435d4a3f0479785a86a027ec3"} Oct 30 20:17:47.121172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:47.121100 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" podStartSLOduration=1.1210779180000001 podStartE2EDuration="1.121077918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:47.11904588 +0000 UTC m=+2534.568538605" watchObservedRunningTime="2026-10-30 20:17:47.121077918 +0000 UTC m=+2534.570570645" Oct 30 20:17:48.099506 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:48.097803 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" event={"ID":"18bb5424-e691-4875-b046-f56eeffe32d4","Type":"ContainerStarted","Data":"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff"} Oct 30 20:17:48.099506 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:48.097893 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:17:48.137400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:48.137301 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" podStartSLOduration=2.137285674 podStartE2EDuration="2.137285674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:48.137255202 +0000 UTC m=+2535.586747927" watchObservedRunningTime="2026-10-30 20:17:48.137285674 +0000 UTC m=+2535.586778399" Oct 30 20:17:50.870553 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:17:50.870511 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:17:51.962515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:51.962447 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:17:51.967651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:51.967627 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:51.984255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:51.984221 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c8fa69-openshift-default-dockercfg-hvqzn\"" Oct 30 20:17:52.000794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.000760 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:17:52.040282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040248 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040448 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4bh\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040491 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040601 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040683 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040740 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040871 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040854 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040887 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.040941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.040906 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142111 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142074 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142193 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4bh\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142220 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142268 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142583 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142436 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142670 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142796 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142698 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142790 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142822 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.142855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.142848 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.143244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.143217 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.143399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.143304 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.143399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.143316 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.145507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.145488 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.145507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.145498 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.162105 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.162071 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.162291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.162214 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4bh\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.284302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.284263 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:52.634770 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.634641 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:17:52.639659 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:17:52.639620 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605ab214_f577_4cd3_aa4b_2794c2c8f21d.slice/crio-8ed9947eb013a6b9af1e2ab21e588ef3dea34333662493d43dc93fbb8c74e27d WatchSource:0}: Error finding container 8ed9947eb013a6b9af1e2ab21e588ef3dea34333662493d43dc93fbb8c74e27d: Status 404 returned error can't find the container with id 8ed9947eb013a6b9af1e2ab21e588ef3dea34333662493d43dc93fbb8c74e27d Oct 30 20:17:52.648276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.648234 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:52.648433 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.648331 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:52.648433 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:52.648397 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:53.146270 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.146229 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" event={"ID":"605ab214-f577-4cd3-aa4b-2794c2c8f21d","Type":"ContainerStarted","Data":"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924"} Oct 30 20:17:53.146872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.146300 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" event={"ID":"605ab214-f577-4cd3-aa4b-2794c2c8f21d","Type":"ContainerStarted","Data":"8ed9947eb013a6b9af1e2ab21e588ef3dea34333662493d43dc93fbb8c74e27d"} Oct 30 20:17:53.189924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.189868 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podStartSLOduration=2.18985316 podStartE2EDuration="2.18985316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:53.188221681 +0000 UTC m=+2540.637714405" watchObservedRunningTime="2026-10-30 20:17:53.18985316 +0000 UTC m=+2540.639345884" Oct 30 20:17:53.285042 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.285006 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:17:53.287109 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.287081 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:53.287263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:53.287138 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:54.290874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:54.290831 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:54.291424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:54.290890 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:54.334158 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:54.334125 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:17:55.287393 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:55.286823 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:55.287393 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:55.286892 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:56.289392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:56.285853 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:56.289392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:56.285938 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:57.288831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:57.288791 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:57.289119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:57.289089 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:58.101053 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:58.101022 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:17:58.288447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:58.287839 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:58.288447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:58.287907 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:17:59.286105 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:59.286062 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:17:59.286617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:17:59.286326 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 20:18:00.114942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:00.114909 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:18:00.291058 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:00.291024 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:18:00.291521 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:00.291497 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:18:00.292401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:00.292380 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:18:01.748969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:01.748933 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:18:01.781030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:01.780988 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:18:06.217230 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.217191 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:18:06.222044 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.222018 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.241734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.241662 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:18:06.249834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.249802 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-188d45-serving-cert\"" Oct 30 20:18:06.316891 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.316850 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.317121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.317015 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.317121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.317064 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.317236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.317219 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqk95\" (UniqueName: \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.418619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.418581 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.418821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.418642 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.418821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.418757 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk95\" (UniqueName: \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.418906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.418856 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.419462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.419434 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.421448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.421420 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.421448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.421443 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.439355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.439315 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk95\" (UniqueName: \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\") pod \"maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.538286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.538248 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:06.750144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.750095 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" podUID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" containerName="payload-processing" containerID="cri-o://4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0" gracePeriod=25 Oct 30 20:18:06.781752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.781697 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" containerID="cri-o://3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c" gracePeriod=25 Oct 30 20:18:06.852082 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:18:06.852051 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc265a09c_4ada_46f6_ab57_8fe19794f506.slice/crio-88a2803da2d3a22e447e177479fcfb548535faade75bea924013ea0cae1ebcc8 WatchSource:0}: Error finding container 88a2803da2d3a22e447e177479fcfb548535faade75bea924013ea0cae1ebcc8: Status 404 returned error can't find the container with id 88a2803da2d3a22e447e177479fcfb548535faade75bea924013ea0cae1ebcc8 Oct 30 20:18:06.853291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:06.853259 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:18:07.024787 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.024759 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:18:07.131278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.129763 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\") pod \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " Oct 30 20:18:07.131278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.130092 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\" (UniqueName: \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\") pod \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\" (UID: \"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1\") " Oct 30 20:18:07.131278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.130594 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume" pod "e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" (UID: "e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:07.131278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.130789 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:07.133334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.133308 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl" pod "e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" (UID: "e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1"). InnerVolumeSpecName "kube-api-access-65dbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:07.232219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.232183 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dbl\" (UniqueName: \"kubernetes.io/projected/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1-kube-api-access-65dbl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:07.300150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.300119 5991 generic.go:356] "Generic (PLEG): container finished" podID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" containerID="4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0" exitCode=0 Oct 30 20:18:07.300311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.300167 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" Oct 30 20:18:07.300311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.300201 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" event={"ID":"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1","Type":"ContainerDied","Data":"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0"} Oct 30 20:18:07.300311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.300262 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5" event={"ID":"e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1","Type":"ContainerDied","Data":"88f54327c6603d2b05c0704e7af318a6eaa9b2ca7cf842294c9bc5250ebdd359"} Oct 30 20:18:07.300311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.300290 5991 scope.go:122] "RemoveContainer" containerID="4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0" Oct 30 20:18:07.302464 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.302423 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" event={"ID":"c265a09c-4ada-46f6-ab57-8fe19794f506","Type":"ContainerStarted","Data":"88a2803da2d3a22e447e177479fcfb548535faade75bea924013ea0cae1ebcc8"} Oct 30 20:18:07.314705 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.314552 5991 scope.go:122] "RemoveContainer" containerID="4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0" Oct 30 20:18:07.318407 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:07.318035 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0\": container with ID starting with 4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0 not found: ID does not exist" containerID="4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0" Oct 30 20:18:07.318407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.318079 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0"} err="failed to get container status \"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0\": rpc error: code = NotFound desc = could not find container \"4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0\": container with ID starting with 4618e04baacf4cca44a10143652227ec4cc565820a89abbeb8a104a7abfd5ad0 not found: ID does not exist" Oct 30 20:18:07.355520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.355481 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:18:07.372894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.372862 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d3c508-6d86497b96-l7bf5"] Oct 30 20:18:07.815308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:07.815272 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:18:08.312453 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:08.312416 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" event={"ID":"c265a09c-4ada-46f6-ab57-8fe19794f506","Type":"ContainerStarted","Data":"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd"} Oct 30 20:18:08.312871 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:08.312495 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:08.353256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:08.353204 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" podStartSLOduration=1.966888312 podStartE2EDuration="2.353191636s" totalImagesPullingTime="386.303324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:08.352296264 +0000 UTC m=+2555.801788987" watchObservedRunningTime="2026-10-30 20:18:08.353191636 +0000 UTC m=+2555.802684358" Oct 30 20:18:08.683314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:08.683217 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" path="/var/lib/kubelet/pods/e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1/volumes" Oct 30 20:18:09.356264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.356223 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" podUID="84bb9568-16fc-4dc7-bf37-71520be39eb8" containerName="main" containerID="cri-o://dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527" gracePeriod=45 Oct 30 20:18:09.689833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.689806 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:18:09.774067 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.773980 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774067 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774039 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774076 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774242 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774293 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\" (UniqueName: \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774293 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:09.774417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774305 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:09.774417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774346 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\") pod \"84bb9568-16fc-4dc7-bf37-71520be39eb8\" (UID: \"84bb9568-16fc-4dc7-bf37-71520be39eb8\") " Oct 30 20:18:09.774604 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.774577 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:09.775159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.775133 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.775304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.775173 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.775304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.775196 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.776553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.776526 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:09.776553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.776545 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:09.776723 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.776676 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq" pod "84bb9568-16fc-4dc7-bf37-71520be39eb8" (UID: "84bb9568-16fc-4dc7-bf37-71520be39eb8"). InnerVolumeSpecName "kube-api-access-7lwgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:09.876655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.876619 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwgq\" (UniqueName: \"kubernetes.io/projected/84bb9568-16fc-4dc7-bf37-71520be39eb8-kube-api-access-7lwgq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.876655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.876651 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bb9568-16fc-4dc7-bf37-71520be39eb8-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.876655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:09.876662 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bb9568-16fc-4dc7-bf37-71520be39eb8-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:10.300687 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.300641 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:10.330539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.330506 5991 generic.go:356] "Generic (PLEG): container finished" podID="84bb9568-16fc-4dc7-bf37-71520be39eb8" containerID="dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527" exitCode=0 Oct 30 20:18:10.330756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.330570 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" Oct 30 20:18:10.330756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.330593 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" event={"ID":"84bb9568-16fc-4dc7-bf37-71520be39eb8","Type":"ContainerDied","Data":"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527"} Oct 30 20:18:10.330756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.330648 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w" event={"ID":"84bb9568-16fc-4dc7-bf37-71520be39eb8","Type":"ContainerDied","Data":"4ea7b5cd52c55b6f6882b2f5b410373e4a3409def9997837865fb1d006223b95"} Oct 30 20:18:10.330756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.330668 5991 scope.go:122] "RemoveContainer" containerID="dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527" Oct 30 20:18:10.342420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.342397 5991 scope.go:122] "RemoveContainer" containerID="dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527" Oct 30 20:18:10.342755 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:10.342733 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527\": container with ID starting with dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527 not found: ID does not exist" containerID="dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527" Oct 30 20:18:10.342815 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.342769 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527"} err="failed to get container status \"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527\": rpc error: code = NotFound desc = could not find container \"dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527\": container with ID starting with dd9a7b36f3aaff6492656b2ee136438ed53a606c1879a622ed27a4ae8d6fe527 not found: ID does not exist" Oct 30 20:18:10.384459 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.384416 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:18:10.396561 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.396524 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a64be2/test-model-a64be2-kserve-5f975c5f5-wnn8w"] Oct 30 20:18:10.680247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:10.680166 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bb9568-16fc-4dc7-bf37-71520be39eb8" path="/var/lib/kubelet/pods/84bb9568-16fc-4dc7-bf37-71520be39eb8/volumes" Oct 30 20:18:12.018664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.018626 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:18:12.019473 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019456 5991 cpu_manager.go:414] "containerMap: removing container" podUID="84bb9568-16fc-4dc7-bf37-71520be39eb8" containerName="main" Oct 30 20:18:12.019473 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019473 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bb9568-16fc-4dc7-bf37-71520be39eb8" containerName="main" Oct 30 20:18:12.019565 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019504 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" containerName="payload-processing" Oct 30 20:18:12.019565 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019509 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" containerName="payload-processing" Oct 30 20:18:12.019626 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019620 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bb9568-16fc-4dc7-bf37-71520be39eb8" containerName="main" Oct 30 20:18:12.019667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.019634 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c02f4a-63b5-4bb9-abd2-3bda6ce7fbc1" containerName="payload-processing" Oct 30 20:18:12.024821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.024791 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.052210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.052174 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9d2720-serving-cert\"" Oct 30 20:18:12.056082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.056043 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:18:12.109456 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.109412 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm72h\" (UniqueName: \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.109646 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.109536 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.109646 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.109596 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.109776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.109752 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.211801 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.211763 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.211985 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.211891 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.212060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.212044 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm72h\" (UniqueName: \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.212156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.212139 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.212567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.212539 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.214451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.214427 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.214615 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.214594 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.232278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.232238 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm72h\" (UniqueName: \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\") pod \"maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.340412 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.340302 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:12.646270 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:12.646240 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:18:12.646496 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:18:12.646476 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963d4ab2_b668_41bc_9ba8_7114f8f38a15.slice/crio-90c48aaf0140b05f004de58bd985b2c22337639ae41908aa16e85f4e68c415b8 WatchSource:0}: Error finding container 90c48aaf0140b05f004de58bd985b2c22337639ae41908aa16e85f4e68c415b8: Status 404 returned error can't find the container with id 90c48aaf0140b05f004de58bd985b2c22337639ae41908aa16e85f4e68c415b8 Oct 30 20:18:13.356390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:13.356278 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" event={"ID":"963d4ab2-b668-41bc-9ba8-7114f8f38a15","Type":"ContainerStarted","Data":"7e6c9b768f7ba6db5e1dc967c88370f4a47c1f1756decc8f5d8d496f4cb074f4"} Oct 30 20:18:13.356390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:13.356333 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" event={"ID":"963d4ab2-b668-41bc-9ba8-7114f8f38a15","Type":"ContainerStarted","Data":"90c48aaf0140b05f004de58bd985b2c22337639ae41908aa16e85f4e68c415b8"} Oct 30 20:18:13.356876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:13.356393 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:13.399187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:13.399129 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" podStartSLOduration=2.013173342 podStartE2EDuration="2.39911409s" totalImagesPullingTime="385.940748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:13.395863568 +0000 UTC m=+2560.845356304" watchObservedRunningTime="2026-10-30 20:18:13.39911409 +0000 UTC m=+2560.848606859" Oct 30 20:18:14.327275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:14.327245 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:18:19.372555 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:19.372515 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:18:20.300658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:20.300621 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:20.870438 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:20.870394 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:18:21.494750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:21.494712 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:18:27.078229 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.078198 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:18:27.150477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.150439 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\") pod \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " Oct 30 20:18:27.150674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.150659 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\" (UniqueName: \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\") pod \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\" (UID: \"dda5ffeb-27fc-4726-ac2c-b2868ee1202b\") " Oct 30 20:18:27.150932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.150894 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume" pod "dda5ffeb-27fc-4726-ac2c-b2868ee1202b" (UID: "dda5ffeb-27fc-4726-ac2c-b2868ee1202b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:27.151223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.151205 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:27.153119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.153093 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w" pod "dda5ffeb-27fc-4726-ac2c-b2868ee1202b" (UID: "dda5ffeb-27fc-4726-ac2c-b2868ee1202b"). InnerVolumeSpecName "kube-api-access-dhf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:27.253184 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.253147 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhf7w\" (UniqueName: \"kubernetes.io/projected/dda5ffeb-27fc-4726-ac2c-b2868ee1202b-kube-api-access-dhf7w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:27.472180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.472150 5991 generic.go:356] "Generic (PLEG): container finished" podID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerID="3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c" exitCode=0 Oct 30 20:18:27.472414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.472195 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" Oct 30 20:18:27.472414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.472241 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" event={"ID":"dda5ffeb-27fc-4726-ac2c-b2868ee1202b","Type":"ContainerDied","Data":"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c"} Oct 30 20:18:27.472414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.472307 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn" event={"ID":"dda5ffeb-27fc-4726-ac2c-b2868ee1202b","Type":"ContainerDied","Data":"58e4530cba9b0387dfc69f3447169149b114ba2b217b9dbb323473a34a4ba14f"} Oct 30 20:18:27.472414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.472336 5991 scope.go:122] "RemoveContainer" containerID="3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c" Oct 30 20:18:27.483882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.483860 5991 scope.go:122] "RemoveContainer" containerID="3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c" Oct 30 20:18:27.484198 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:27.484179 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c\": container with ID starting with 3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c not found: ID does not exist" containerID="3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c" Oct 30 20:18:27.484251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.484208 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c"} err="failed to get container status \"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c\": rpc error: code = NotFound desc = could not find container \"3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c\": container with ID starting with 3cd3fae45576c43de7954bf28a10c3273d879cc054f7cfd8c51d93c07ff39c1c not found: ID does not exist" Oct 30 20:18:27.523944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.523905 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:18:27.535776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.535737 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d3c508-7ff8cb799f-dlsnn"] Oct 30 20:18:27.975485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.975450 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 20:18:27.975682 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:27.975650 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" containerID="cri-o://379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617" gracePeriod=30 Oct 30 20:18:28.680587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:28.680549 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" path="/var/lib/kubelet/pods/dda5ffeb-27fc-4726-ac2c-b2868ee1202b/volumes" Oct 30 20:18:31.987603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:31.987556 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:18:33.278081 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.278050 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 20:18:33.343982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.343952 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344143 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344083 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344203 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344144 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344210 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344321 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344458 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344397 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:33.344458 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344434 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344487 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:33.344563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344494 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.344679 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344555 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344848 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.344852 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\") pod \"204e3272-eecd-4153-80b1-5b628e1221cf\" (UID: \"204e3272-eecd-4153-80b1-5b628e1221cf\") " Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345035 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345664 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345686 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345701 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e3272-eecd-4153-80b1-5b628e1221cf-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345715 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.345790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.345727 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.347447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.347423 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "kube-api-access-75kmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:33.347586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.347452 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:33.351461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.351434 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:33.359519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.359489 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token" pod "204e3272-eecd-4153-80b1-5b628e1221cf" (UID: "204e3272-eecd-4153-80b1-5b628e1221cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:33.447098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.447062 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.447098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.447095 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kmp\" (UniqueName: \"kubernetes.io/projected/204e3272-eecd-4153-80b1-5b628e1221cf-kube-api-access-75kmp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.447323 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.447113 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e3272-eecd-4153-80b1-5b628e1221cf-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.447323 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.447123 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e3272-eecd-4153-80b1-5b628e1221cf-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:33.517978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.517893 5991 generic.go:356] "Generic (PLEG): container finished" podID="204e3272-eecd-4153-80b1-5b628e1221cf" containerID="379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617" exitCode=0 Oct 30 20:18:33.517978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.517942 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" Oct 30 20:18:33.517978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.517951 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" event={"ID":"204e3272-eecd-4153-80b1-5b628e1221cf","Type":"ContainerDied","Data":"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617"} Oct 30 20:18:33.518228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.518010 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf" event={"ID":"204e3272-eecd-4153-80b1-5b628e1221cf","Type":"ContainerDied","Data":"7de6ab9db4a4c17f16eb8cb6341d5a1ea452660880b8e62ac2924ec2ef9c79c8"} Oct 30 20:18:33.518228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.518040 5991 scope.go:122] "RemoveContainer" containerID="379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617" Oct 30 20:18:33.529598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.529578 5991 scope.go:122] "RemoveContainer" containerID="379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617" Oct 30 20:18:33.529867 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:33.529850 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617\": container with ID starting with 379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617 not found: ID does not exist" containerID="379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617" Oct 30 20:18:33.529926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.529876 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617"} err="failed to get container status \"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617\": rpc error: code = NotFound desc = could not find container \"379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617\": container with ID starting with 379c1c3d4be044d0268e6ff9a4163961f05a9b73df753db1ee639d4e251f4617 not found: ID does not exist" Oct 30 20:18:33.571700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.571665 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 20:18:33.579758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:33.579731 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lq4zf"] Oct 30 20:18:34.680218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:34.680119 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" path="/var/lib/kubelet/pods/204e3272-eecd-4153-80b1-5b628e1221cf/volumes" Oct 30 20:18:36.516539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.516498 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" podUID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" containerName="main" containerID="cri-o://92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9" gracePeriod=45 Oct 30 20:18:36.795846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.795821 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:18:36.894952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.894904 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.895161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.895036 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\" (UniqueName: \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.895254 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.895230 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.895325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.895289 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.895403 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.895388 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.895470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.895435 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\") pod \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\" (UID: \"17b2ff98-28ad-430a-b5cc-85281ffe12dc\") " Oct 30 20:18:36.896799 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.896445 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:36.897700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.897626 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:36.897700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.897651 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:36.899512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.899491 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:36.900147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.900121 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "kube-api-access-2nl9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:36.902958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.902930 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs" pod "17b2ff98-28ad-430a-b5cc-85281ffe12dc" (UID: "17b2ff98-28ad-430a-b5cc-85281ffe12dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:36.988474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.988346 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" containerID="cri-o://e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1" gracePeriod=25 Oct 30 20:18:36.997909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997867 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:36.997909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997905 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:36.997909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997915 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:36.998161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997923 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:36.998161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997931 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2ff98-28ad-430a-b5cc-85281ffe12dc-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:36.998161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:36.997940 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nl9z\" (UniqueName: \"kubernetes.io/projected/17b2ff98-28ad-430a-b5cc-85281ffe12dc-kube-api-access-2nl9z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:37.555464 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.555429 5991 generic.go:356] "Generic (PLEG): container finished" podID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" containerID="92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9" exitCode=0 Oct 30 20:18:37.555934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.555480 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" event={"ID":"17b2ff98-28ad-430a-b5cc-85281ffe12dc","Type":"ContainerDied","Data":"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9"} Oct 30 20:18:37.555934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.555491 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" Oct 30 20:18:37.555934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.555512 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x" event={"ID":"17b2ff98-28ad-430a-b5cc-85281ffe12dc","Type":"ContainerDied","Data":"b9f0878396f077ee0f8e42af89b0458a9978a02a761c7f5e1bc3c49c0635126d"} Oct 30 20:18:37.555934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.555528 5991 scope.go:122] "RemoveContainer" containerID="92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9" Oct 30 20:18:37.567812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.567786 5991 scope.go:122] "RemoveContainer" containerID="92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9" Oct 30 20:18:37.568107 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:37.568087 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9\": container with ID starting with 92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9 not found: ID does not exist" containerID="92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9" Oct 30 20:18:37.568170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.568115 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9"} err="failed to get container status \"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9\": rpc error: code = NotFound desc = could not find container \"92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9\": container with ID starting with 92209a9510d5f7479ad719779fcb81a9c9a4fdff4797301743c3ec4c1b69e4c9 not found: ID does not exist" Oct 30 20:18:37.610957 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.610923 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:18:37.623328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:37.623293 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4cfcee/test-model-4cfcee-kserve-74fd4f447f-bg64x"] Oct 30 20:18:38.684436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:38.684395 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" path="/var/lib/kubelet/pods/17b2ff98-28ad-430a-b5cc-85281ffe12dc/volumes" Oct 30 20:18:40.522172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:40.522137 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:41.765256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:41.765208 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:18:41.765769 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:41.765486 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" containerID="cri-o://a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38" gracePeriod=30 Oct 30 20:18:47.058585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.058554 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:18:47.149107 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149076 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149174 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149230 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149294 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149423 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149466 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149507 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:47.149784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149626 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149660 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149758 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\") pod \"c10d707c-0645-43dc-a977-84d82fd1478c\" (UID: \"c10d707c-0645-43dc-a977-84d82fd1478c\") " Oct 30 20:18:47.149784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.149761 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:47.150289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.150267 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:47.150538 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.150425 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.150704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.150689 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.151033 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.151008 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:47.151375 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.151343 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:47.152965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.152940 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:47.153758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.153614 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:47.153972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.153905 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "kube-api-access-k5vd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:47.154584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.154552 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo" pod "c10d707c-0645-43dc-a977-84d82fd1478c" (UID: "c10d707c-0645-43dc-a977-84d82fd1478c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:47.252031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.251995 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vd4\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-kube-api-access-k5vd4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252029 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252031 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252041 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252050 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d707c-0645-43dc-a977-84d82fd1478c-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252058 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d707c-0645-43dc-a977-84d82fd1478c-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252065 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d707c-0645-43dc-a977-84d82fd1478c-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.252298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.252072 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d707c-0645-43dc-a977-84d82fd1478c-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:47.634823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.634793 5991 generic.go:356] "Generic (PLEG): container finished" podID="c10d707c-0645-43dc-a977-84d82fd1478c" containerID="a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38" exitCode=0 Oct 30 20:18:47.635028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.634843 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" Oct 30 20:18:47.635028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.634880 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" event={"ID":"c10d707c-0645-43dc-a977-84d82fd1478c","Type":"ContainerDied","Data":"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38"} Oct 30 20:18:47.635028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.634935 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn" event={"ID":"c10d707c-0645-43dc-a977-84d82fd1478c","Type":"ContainerDied","Data":"5f1a61e36236c762d8a08b5e3f9bb8a39231dc8b7ef49d0fc189f13b3fed1093"} Oct 30 20:18:47.635028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.634963 5991 scope.go:122] "RemoveContainer" containerID="a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38" Oct 30 20:18:47.647309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.647289 5991 scope.go:122] "RemoveContainer" containerID="a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38" Oct 30 20:18:47.647716 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:47.647696 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38\": container with ID starting with a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38 not found: ID does not exist" containerID="a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38" Oct 30 20:18:47.647802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.647730 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38"} err="failed to get container status \"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38\": rpc error: code = NotFound desc = could not find container \"a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38\": container with ID starting with a2d05d8a74f073c1d95a686d2f6149237d54fb67c69ecdc7f06bcfb47abe7d38 not found: ID does not exist" Oct 30 20:18:47.700272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.700235 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:18:47.727666 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:47.727625 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-sj2wn"] Oct 30 20:18:48.680470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:48.680435 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" path="/var/lib/kubelet/pods/c10d707c-0645-43dc-a977-84d82fd1478c/volumes" Oct 30 20:18:50.521685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:50.521641 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:50.870465 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:50.870358 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:18:57.279108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.279078 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:18:57.395684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.395647 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\" (UniqueName: \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\") pod \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " Oct 30 20:18:57.395901 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.395773 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\") pod \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\" (UID: \"f0ab3c7a-394a-4c1e-a29d-454df4a5766e\") " Oct 30 20:18:57.396252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.396219 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume" pod "f0ab3c7a-394a-4c1e-a29d-454df4a5766e" (UID: "f0ab3c7a-394a-4c1e-a29d-454df4a5766e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:57.398002 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.397976 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj" pod "f0ab3c7a-394a-4c1e-a29d-454df4a5766e" (UID: "f0ab3c7a-394a-4c1e-a29d-454df4a5766e"). InnerVolumeSpecName "kube-api-access-t2dvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:57.497914 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.497879 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dvj\" (UniqueName: \"kubernetes.io/projected/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-kube-api-access-t2dvj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:57.497914 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.497911 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ab3c7a-394a-4c1e-a29d-454df4a5766e-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:18:57.708487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.708392 5991 generic.go:356] "Generic (PLEG): container finished" podID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerID="e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1" exitCode=0 Oct 30 20:18:57.708487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.708452 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" Oct 30 20:18:57.708487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.708459 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" event={"ID":"f0ab3c7a-394a-4c1e-a29d-454df4a5766e","Type":"ContainerDied","Data":"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1"} Oct 30 20:18:57.708743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.708516 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92" event={"ID":"f0ab3c7a-394a-4c1e-a29d-454df4a5766e","Type":"ContainerDied","Data":"c864d6df491892ded3767273bdb3e8ecf2a4eed47ec40907aa79377790728781"} Oct 30 20:18:57.708743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.708538 5991 scope.go:122] "RemoveContainer" containerID="e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1" Oct 30 20:18:57.719951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.719930 5991 scope.go:122] "RemoveContainer" containerID="e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1" Oct 30 20:18:57.720292 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:18:57.720266 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1\": container with ID starting with e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1 not found: ID does not exist" containerID="e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1" Oct 30 20:18:57.720401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.720304 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1"} err="failed to get container status \"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1\": rpc error: code = NotFound desc = could not find container \"e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1\": container with ID starting with e63e7440e5d77e7fd2abd1fd38a9f6606d15f4216283097d885100d870dce3a1 not found: ID does not exist" Oct 30 20:18:57.762593 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.762555 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:18:57.765686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:57.765661 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a64be2-5cd974cd7f-6bn92"] Oct 30 20:18:58.680357 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:18:58.680324 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" path="/var/lib/kubelet/pods/f0ab3c7a-394a-4c1e-a29d-454df4a5766e/volumes" Oct 30 20:19:20.872071 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:19:20.871984 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:19:27.155210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.155175 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:19:27.155805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.155423 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" podUID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" containerName="maas-api" containerID="cri-o://bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6" gracePeriod=30 Oct 30 20:19:27.469446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.469416 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:19:27.529638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.529598 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\") pod \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " Oct 30 20:19:27.529835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.529674 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\") pod \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " Oct 30 20:19:27.529922 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.529854 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\") pod \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " Oct 30 20:19:27.529985 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.529915 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\" (UniqueName: \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\") pod \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\" (UID: \"a2eff3d8-9984-478f-a4e8-98b23b885c2a\") " Oct 30 20:19:27.530154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.530110 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca" pod "a2eff3d8-9984-478f-a4e8-98b23b885c2a" (UID: "a2eff3d8-9984-478f-a4e8-98b23b885c2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:27.530579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.530560 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eff3d8-9984-478f-a4e8-98b23b885c2a-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:19:27.535409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.532445 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs" pod "a2eff3d8-9984-478f-a4e8-98b23b885c2a" (UID: "a2eff3d8-9984-478f-a4e8-98b23b885c2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:27.535409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.532490 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls" pod "a2eff3d8-9984-478f-a4e8-98b23b885c2a" (UID: "a2eff3d8-9984-478f-a4e8-98b23b885c2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:27.535409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.532770 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk" pod "a2eff3d8-9984-478f-a4e8-98b23b885c2a" (UID: "a2eff3d8-9984-478f-a4e8-98b23b885c2a"). InnerVolumeSpecName "kube-api-access-rgbjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:27.632511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.632472 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:19:27.632511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.632511 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbjk\" (UniqueName: \"kubernetes.io/projected/a2eff3d8-9984-478f-a4e8-98b23b885c2a-kube-api-access-rgbjk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:19:27.632751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.632526 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eff3d8-9984-478f-a4e8-98b23b885c2a-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:19:27.965356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.965310 5991 generic.go:356] "Generic (PLEG): container finished" podID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" containerID="bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6" exitCode=0 Oct 30 20:19:27.965589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.965378 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" event={"ID":"a2eff3d8-9984-478f-a4e8-98b23b885c2a","Type":"ContainerDied","Data":"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6"} Oct 30 20:19:27.965589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.965425 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" Oct 30 20:19:27.965589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.965449 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh" event={"ID":"a2eff3d8-9984-478f-a4e8-98b23b885c2a","Type":"ContainerDied","Data":"2d6023f322b2b355bf871e636ad92e4670c9fbb8ef801f59b0e5be0d062a2da5"} Oct 30 20:19:27.965589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.965485 5991 scope.go:122] "RemoveContainer" containerID="bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6" Oct 30 20:19:27.978028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.978010 5991 scope.go:122] "RemoveContainer" containerID="bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6" Oct 30 20:19:27.978387 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:19:27.978351 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6\": container with ID starting with bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6 not found: ID does not exist" containerID="bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6" Oct 30 20:19:27.978447 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:27.978398 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6"} err="failed to get container status \"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6\": rpc error: code = NotFound desc = could not find container \"bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6\": container with ID starting with bbd94c5409750a692ca662164357d760642a228844b015a38ee9d5f585a1cda6 not found: ID does not exist" Oct 30 20:19:28.017262 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:28.017225 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:19:28.029799 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:28.029766 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cfcee-5458fbff4c-v5xxh"] Oct 30 20:19:28.680432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:28.680392 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" path="/var/lib/kubelet/pods/a2eff3d8-9984-478f-a4e8-98b23b885c2a/volumes" Oct 30 20:19:29.080429 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:29.080333 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:19:32.101546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.101509 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:19:32.103183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.103152 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" Oct 30 20:19:32.103392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.103354 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" Oct 30 20:19:32.104434 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104416 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" Oct 30 20:19:32.104556 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104543 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" Oct 30 20:19:32.104684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104673 5991 cpu_manager.go:414] "containerMap: removing container" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" Oct 30 20:19:32.104763 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104752 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" Oct 30 20:19:32.104863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104853 5991 cpu_manager.go:414] "containerMap: removing container" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" Oct 30 20:19:32.104947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.104937 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" Oct 30 20:19:32.105024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105015 5991 cpu_manager.go:414] "containerMap: removing container" podUID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" containerName="main" Oct 30 20:19:32.105096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105086 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" containerName="main" Oct 30 20:19:32.105179 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105169 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" containerName="maas-api" Oct 30 20:19:32.105250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105240 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" containerName="maas-api" Oct 30 20:19:32.105592 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105560 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10d707c-0645-43dc-a977-84d82fd1478c" containerName="istio-proxy" Oct 30 20:19:32.105862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.105833 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda5ffeb-27fc-4726-ac2c-b2868ee1202b" containerName="payload-pre-processing" Oct 30 20:19:32.106019 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.106005 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="204e3272-eecd-4153-80b1-5b628e1221cf" containerName="istio-proxy" Oct 30 20:19:32.106145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.106123 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b2ff98-28ad-430a-b5cc-85281ffe12dc" containerName="main" Oct 30 20:19:32.106240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.106223 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2eff3d8-9984-478f-a4e8-98b23b885c2a" containerName="maas-api" Oct 30 20:19:32.106395 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.106254 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ab3c7a-394a-4c1e-a29d-454df4a5766e" containerName="payload-pre-processing" Oct 30 20:19:32.110733 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.110709 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.133738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.133688 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-188d45\"" Oct 30 20:19:32.134813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.134784 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:19:32.199681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.199640 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.199906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.199772 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbmk\" (UniqueName: \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.301722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.301677 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.301926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.301806 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbmk\" (UniqueName: \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.302487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.302463 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.321436 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.321407 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbmk\" (UniqueName: \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\") pod \"payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.440808 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.440722 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:32.752338 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:19:32.752305 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed9b983_3fc1_487a_9cd0_2a4660f5db7f.slice/crio-eb5aa57049badba7501434eda77184c528c3b82de872583b00d916e022857dec WatchSource:0}: Error finding container eb5aa57049badba7501434eda77184c528c3b82de872583b00d916e022857dec: Status 404 returned error can't find the container with id eb5aa57049badba7501434eda77184c528c3b82de872583b00d916e022857dec Oct 30 20:19:32.752674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:32.752650 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:19:33.004601 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:33.004511 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" event={"ID":"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f","Type":"ContainerStarted","Data":"eb5aa57049badba7501434eda77184c528c3b82de872583b00d916e022857dec"} Oct 30 20:19:35.020974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:35.020931 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" event={"ID":"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f","Type":"ContainerStarted","Data":"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6"} Oct 30 20:19:35.021545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:35.021004 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:35.062998 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:35.062936 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" podStartSLOduration=1.817032122 podStartE2EDuration="3.062918621s" totalImagesPullingTime="1.245886499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:35.060656245 +0000 UTC m=+2642.510148970" watchObservedRunningTime="2026-10-30 20:19:35.062918621 +0000 UTC m=+2642.512411345" Oct 30 20:19:36.987309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:36.987274 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:19:36.994108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:36.994081 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.011785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.011750 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6175d5-openshift-default-dockercfg-srzcz\"" Oct 30 20:19:37.029593 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.029555 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:19:37.070011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.069972 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070046 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070136 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070156 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjbg\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070238 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070293 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070426 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070468 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.070651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.070635 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.171840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.171787 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.171968 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172020 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172138 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172228 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172216 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172286 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172333 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172513 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172355 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjbg\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172513 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172398 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172513 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172439 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.172832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.172813 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.173280 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.173262 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.174198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.174169 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.174507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.174484 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.176299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.176264 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.176992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.176964 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.203559 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.200267 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.211418 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.208084 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjbg\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\") pod \"e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.311048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.310954 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:37.638560 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:19:37.638529 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfb0705_0c49_4e44_a2af_f3c042ad6662.slice/crio-d9972c206bbd093244f252dcb205d865eb161a2ab36beed2819297bcac70184c WatchSource:0}: Error finding container d9972c206bbd093244f252dcb205d865eb161a2ab36beed2819297bcac70184c: Status 404 returned error can't find the container with id d9972c206bbd093244f252dcb205d865eb161a2ab36beed2819297bcac70184c Oct 30 20:19:37.638933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.638909 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:19:37.641663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.641629 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:37.641764 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.641699 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:37.641764 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:37.641728 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:38.052809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:38.052770 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerStarted","Data":"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e"} Oct 30 20:19:38.053268 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:38.052834 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerStarted","Data":"d9972c206bbd093244f252dcb205d865eb161a2ab36beed2819297bcac70184c"} Oct 30 20:19:38.312225 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:38.312117 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:38.318463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:38.318427 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:38.366995 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:38.366926 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podStartSLOduration=2.366911875 podStartE2EDuration="2.366911875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:38.119164475 +0000 UTC m=+2645.568657201" watchObservedRunningTime="2026-10-30 20:19:38.366911875 +0000 UTC m=+2645.816404693" Oct 30 20:19:39.060840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:39.060803 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:39.061840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:39.061817 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:19:46.033779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:46.033748 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:19:50.870449 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:19:50.870400 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:19:54.784819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.784781 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:19:54.790816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.790785 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.810190 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.810154 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:19:54.817326 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.817291 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-159b3a\"/\"sub-test-model-159b3a-kserve-self-signed-certs\"" Oct 30 20:19:54.817558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.817430 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-159b3a\"/\"default-dockercfg-28hkv\"" Oct 30 20:19:54.818896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.818869 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-159b3a\"/\"openshift-service-ca.crt\"" Oct 30 20:19:54.819020 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.818949 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-159b3a\"/\"kube-root-ca.crt\"" Oct 30 20:19:54.939674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.939642 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.939872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.939792 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.939949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.939906 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8d8\" (UniqueName: \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.940074 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.940049 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.940214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.940092 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:54.940214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:54.940185 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.043667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.043556 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044139 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044057 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044060 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044273 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044237 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8d8\" (UniqueName: \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044456 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044351 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044456 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044423 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.044589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044474 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.045804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.044804 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.047091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.047068 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.049794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.049749 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.068739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.068699 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8d8\" (UniqueName: \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.077735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.077695 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\") pod \"sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.111165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.111121 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:55.436744 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:19:55.436707 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124bee40_600e_4e23_8487_fff9803301c0.slice/crio-42a9fb0b05e71fb90f433701a0512efc259e00b9059edd613374f6cc1c91d17f WatchSource:0}: Error finding container 42a9fb0b05e71fb90f433701a0512efc259e00b9059edd613374f6cc1c91d17f: Status 404 returned error can't find the container with id 42a9fb0b05e71fb90f433701a0512efc259e00b9059edd613374f6cc1c91d17f Oct 30 20:19:55.447606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:55.440858 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:19:56.213973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:56.213934 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" event={"ID":"124bee40-600e-4e23-8487-fff9803301c0","Type":"ContainerStarted","Data":"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf"} Oct 30 20:19:56.214432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:56.213980 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" event={"ID":"124bee40-600e-4e23-8487-fff9803301c0","Type":"ContainerStarted","Data":"42a9fb0b05e71fb90f433701a0512efc259e00b9059edd613374f6cc1c91d17f"} Oct 30 20:19:56.254737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:56.254508 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" podStartSLOduration=2.254489371 podStartE2EDuration="2.254489371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:56.254167877 +0000 UTC m=+2663.703660602" watchObservedRunningTime="2026-10-30 20:19:56.254489371 +0000 UTC m=+2663.703982095" Oct 30 20:19:57.111930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.111885 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:19:57.310050 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.310014 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:19:57.317289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.317257 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.332686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.332651 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:19:57.360481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.360443 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30e92c\"" Oct 30 20:19:57.360701 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.360622 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30e92c-dockercfg-85fkb\"" Oct 30 20:19:57.495435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.491514 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.495435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.491839 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsck\" (UniqueName: \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.593313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.593276 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.593540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.593525 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsck\" (UniqueName: \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.594054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.594031 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.614790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.614759 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsck\" (UniqueName: \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.635542 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.635507 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:57.948037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:57.948003 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:19:57.948811 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:19:57.948778 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff29d67_ff6d_4f60_bca3_b554126bf08c.slice/crio-5536d694723c957b663e908fbd125f1083eab8123c19e7a3423b6477346f0fc6 WatchSource:0}: Error finding container 5536d694723c957b663e908fbd125f1083eab8123c19e7a3423b6477346f0fc6: Status 404 returned error can't find the container with id 5536d694723c957b663e908fbd125f1083eab8123c19e7a3423b6477346f0fc6 Oct 30 20:19:58.231947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:58.231898 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" event={"ID":"8ff29d67-ff6d-4f60-bca3-b554126bf08c","Type":"ContainerStarted","Data":"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337"} Oct 30 20:19:58.232269 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:58.231961 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" event={"ID":"8ff29d67-ff6d-4f60-bca3-b554126bf08c","Type":"ContainerStarted","Data":"5536d694723c957b663e908fbd125f1083eab8123c19e7a3423b6477346f0fc6"} Oct 30 20:19:58.232269 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:58.231986 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:19:58.273254 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:19:58.273195 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" podStartSLOduration=1.273179005 podStartE2EDuration="1.273179005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:58.27249344 +0000 UTC m=+2665.721986165" watchObservedRunningTime="2026-10-30 20:19:58.273179005 +0000 UTC m=+2665.722671729" Oct 30 20:20:04.009060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.009023 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:20:04.013820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.013789 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.042938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.041471 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"auth-test-model-9d2720-kserve-self-signed-certs\"" Oct 30 20:20:04.042938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.041469 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"default-dockercfg-6dq9g\"" Oct 30 20:20:04.042938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.041543 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"kube-root-ca.crt\"" Oct 30 20:20:04.042938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.041616 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"openshift-service-ca.crt\"" Oct 30 20:20:04.045578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.045547 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:20:04.086748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.086710 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.086940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.086842 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj278\" (UniqueName: \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.086940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.086926 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.087025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.086999 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.087062 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.087029 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.087096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.087083 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.188919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.188877 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.188944 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189117 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189029 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189128 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189280 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189218 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj278\" (UniqueName: \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189304 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189396 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189442 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.189727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.189705 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.191542 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.191512 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.191684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.191665 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.210135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.210098 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj278\" (UniqueName: \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\") pod \"auth-test-model-9d2720-kserve-77bd5f6cc7-72htn\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.328835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.328743 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:04.642168 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:20:04.642133 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod915aea68_342e_4af2_9ea6_2f5d1ed3b84e.slice/crio-34da27c22244a4a424f6e3faa7b6db878694cd5d7086faf095ec54a6abcc2e82 WatchSource:0}: Error finding container 34da27c22244a4a424f6e3faa7b6db878694cd5d7086faf095ec54a6abcc2e82: Status 404 returned error can't find the container with id 34da27c22244a4a424f6e3faa7b6db878694cd5d7086faf095ec54a6abcc2e82 Oct 30 20:20:04.642350 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:04.642324 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:20:05.112045 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.112007 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:20:05.125800 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.125768 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:20:05.286239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.286201 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" event={"ID":"915aea68-342e-4af2-9ea6-2f5d1ed3b84e","Type":"ContainerStarted","Data":"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9"} Oct 30 20:20:05.286239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.286245 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" event={"ID":"915aea68-342e-4af2-9ea6-2f5d1ed3b84e","Type":"ContainerStarted","Data":"34da27c22244a4a424f6e3faa7b6db878694cd5d7086faf095ec54a6abcc2e82"} Oct 30 20:20:05.297906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.297864 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:20:05.329936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.329898 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:05.330780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:05.330723 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" podStartSLOduration=2.330707025 podStartE2EDuration="2.330707025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:05.32748974 +0000 UTC m=+2672.776982465" watchObservedRunningTime="2026-10-30 20:20:05.330707025 +0000 UTC m=+2672.780200147" Oct 30 20:20:09.242668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:09.242637 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:20:14.329339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:14.329300 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:14.343159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:14.343129 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:14.369755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:14.369720 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:20:20.880488 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:20:20.880445 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:20:33.197341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:33.197307 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:20:33.205783 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:33.205751 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:20:36.544665 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.544628 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:20:36.549760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.549731 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.567825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.567789 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-30e92c-serving-cert\"" Oct 30 20:20:36.585416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.585380 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:20:36.596383 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.596329 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.596587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.596495 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.596723 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.596678 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.596862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.596731 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbhk\" (UniqueName: \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.698169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.698126 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.698169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.698170 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbhk\" (UniqueName: \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.698427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.698314 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.698427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.698352 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.699192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.699167 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.701014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.700987 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.701148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.701059 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.719539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.719508 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbhk\" (UniqueName: \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\") pod \"maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:36.869518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:36.869422 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:37.185203 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:37.185173 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:20:37.185356 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:20:37.185181 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13bb142_e034_4f7c_8a54_fdc3c7cab4ae.slice/crio-1f79ea5377d3aeb3dcfef07110dc37faf5f36ece9315f75e521c04732f39f79d WatchSource:0}: Error finding container 1f79ea5377d3aeb3dcfef07110dc37faf5f36ece9315f75e521c04732f39f79d: Status 404 returned error can't find the container with id 1f79ea5377d3aeb3dcfef07110dc37faf5f36ece9315f75e521c04732f39f79d Oct 30 20:20:37.534767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:37.534728 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" event={"ID":"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae","Type":"ContainerStarted","Data":"1f79ea5377d3aeb3dcfef07110dc37faf5f36ece9315f75e521c04732f39f79d"} Oct 30 20:20:38.546959 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:38.546913 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" event={"ID":"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae","Type":"ContainerStarted","Data":"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28"} Oct 30 20:20:38.547454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:38.547179 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:38.589655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:38.589571 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" podStartSLOduration=2.25595149 podStartE2EDuration="2.589548508s" totalImagesPullingTime="333.597018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:38.589112191 +0000 UTC m=+2706.038604911" watchObservedRunningTime="2026-10-30 20:20:38.589548508 +0000 UTC m=+2706.039041231" Oct 30 20:20:44.561337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:44.561251 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:20:45.876085 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:45.876039 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:20:50.897067 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:20:50.897029 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:20:58.740498 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.740463 5991 generic.go:356] "Generic (PLEG): container finished" podID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerID="1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e" exitCode=0 Oct 30 20:20:58.741080 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.740541 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerDied","Data":"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e"} Oct 30 20:20:58.741156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.741138 5991 scope.go:122] "RemoveContainer" containerID="1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e" Oct 30 20:20:58.743035 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.743004 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:58.743133 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.743076 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:58.743133 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:58.743109 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:59.749953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:20:59.749916 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerStarted","Data":"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c"} Oct 30 20:21:00.311657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:00.311620 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:21:00.311913 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:00.311880 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:00.311976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:00.311949 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:01.311967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:01.311928 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:01.312416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:01.311982 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:02.312049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:02.312005 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:02.312537 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:02.312078 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:03.312427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:03.312352 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:03.312870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:03.312461 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:04.311354 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:04.311309 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:04.311615 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:04.311406 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:05.311594 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:05.311553 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:21:05.312010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:05.311627 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 20:21:06.317252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:06.317218 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:21:06.317760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:06.317735 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:21:06.318459 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:06.318442 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:21:20.870533 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:21:20.870478 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:21:22.180589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:22.180549 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:21:27.181307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.181257 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" podUID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" containerName="payload-pre-processing" containerID="cri-o://ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337" gracePeriod=25 Oct 30 20:21:27.475405 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.475377 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:21:27.630595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.630550 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\" (UniqueName: \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\") pod \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " Oct 30 20:21:27.630809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.630609 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\") pod \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\" (UID: \"8ff29d67-ff6d-4f60-bca3-b554126bf08c\") " Oct 30 20:21:27.631025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.630998 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume" pod "8ff29d67-ff6d-4f60-bca3-b554126bf08c" (UID: "8ff29d67-ff6d-4f60-bca3-b554126bf08c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:27.631259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.631243 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff29d67-ff6d-4f60-bca3-b554126bf08c-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:21:27.632865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.632842 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck" pod "8ff29d67-ff6d-4f60-bca3-b554126bf08c" (UID: "8ff29d67-ff6d-4f60-bca3-b554126bf08c"). InnerVolumeSpecName "kube-api-access-mdsck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:27.732852 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.732803 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdsck\" (UniqueName: \"kubernetes.io/projected/8ff29d67-ff6d-4f60-bca3-b554126bf08c-kube-api-access-mdsck\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:21:27.992238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.992147 5991 generic.go:356] "Generic (PLEG): container finished" podID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" containerID="ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337" exitCode=0 Oct 30 20:21:27.992238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.992199 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" Oct 30 20:21:27.992238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.992228 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" event={"ID":"8ff29d67-ff6d-4f60-bca3-b554126bf08c","Type":"ContainerDied","Data":"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337"} Oct 30 20:21:27.992539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.992285 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l" event={"ID":"8ff29d67-ff6d-4f60-bca3-b554126bf08c","Type":"ContainerDied","Data":"5536d694723c957b663e908fbd125f1083eab8123c19e7a3423b6477346f0fc6"} Oct 30 20:21:27.992539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:27.992309 5991 scope.go:122] "RemoveContainer" containerID="ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337" Oct 30 20:21:28.004356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:28.004334 5991 scope.go:122] "RemoveContainer" containerID="ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337" Oct 30 20:21:28.004707 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:21:28.004684 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337\": container with ID starting with ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337 not found: ID does not exist" containerID="ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337" Oct 30 20:21:28.004752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:28.004716 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337"} err="failed to get container status \"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337\": rpc error: code = NotFound desc = could not find container \"ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337\": container with ID starting with ee5bf8f3b72d9fc4ef10ce9613803aa955e395ac6c7da52de18fff78c072a337 not found: ID does not exist" Oct 30 20:21:28.047465 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:28.047430 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:21:28.058892 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:28.058858 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-b6dffb459-nmm2l"] Oct 30 20:21:28.684102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:28.684058 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" path="/var/lib/kubelet/pods/8ff29d67-ff6d-4f60-bca3-b554126bf08c/volumes" Oct 30 20:21:34.459427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:34.459386 5991 scope.go:122] "RemoveContainer" containerID="d184521506d5ad223f2617fba83dc92478b7f9f9f7e31c5b86a540c749b94e02" Oct 30 20:21:45.540480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:45.540446 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:21:46.759098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.759064 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:21:46.760039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.760014 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" containerName="payload-pre-processing" Oct 30 20:21:46.760152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.760040 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" containerName="payload-pre-processing" Oct 30 20:21:46.760344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.760328 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff29d67-ff6d-4f60-bca3-b554126bf08c" containerName="payload-pre-processing" Oct 30 20:21:46.763864 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.763840 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:46.796068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.792601 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6175d5\"" Oct 30 20:21:46.796068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.792626 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6175d5-dockercfg-8rx8f\"" Oct 30 20:21:46.797924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.797893 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:21:46.927471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.927433 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:46.927652 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:46.927594 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltlh\" (UniqueName: \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.028799 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.028706 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.028974 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.028828 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltlh\" (UniqueName: \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.029587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.029564 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.049479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.049436 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltlh\" (UniqueName: \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.078960 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.078922 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:47.385183 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:21:47.385147 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dff2bea_e37c_450e_98ad_109c2b70f6f3.slice/crio-5eca8c52f5b54fa473e114222f04ec2497b6f00f7ae13095b889871054d0c9cf WatchSource:0}: Error finding container 5eca8c52f5b54fa473e114222f04ec2497b6f00f7ae13095b889871054d0c9cf: Status 404 returned error can't find the container with id 5eca8c52f5b54fa473e114222f04ec2497b6f00f7ae13095b889871054d0c9cf Oct 30 20:21:47.386635 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:47.386608 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:21:48.144869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:48.144824 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" event={"ID":"9dff2bea-e37c-450e-98ad-109c2b70f6f3","Type":"ContainerStarted","Data":"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b"} Oct 30 20:21:48.145381 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:48.144884 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" event={"ID":"9dff2bea-e37c-450e-98ad-109c2b70f6f3","Type":"ContainerStarted","Data":"5eca8c52f5b54fa473e114222f04ec2497b6f00f7ae13095b889871054d0c9cf"} Oct 30 20:21:48.145381 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:48.144921 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:21:48.190779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:48.190723 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" podStartSLOduration=2.190710747 podStartE2EDuration="2.190710747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:48.187538958 +0000 UTC m=+2775.637031684" watchObservedRunningTime="2026-10-30 20:21:48.190710747 +0000 UTC m=+2775.640203472" Oct 30 20:21:50.870932 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:21:50.870889 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:21:59.154083 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:21:59.154054 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:22:05.281976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.281933 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:22:05.287919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.287840 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.300977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.300943 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:22:05.307691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.307013 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30e92c\"" Oct 30 20:22:05.307691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.307154 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30e92c-dockercfg-8xkr9\"" Oct 30 20:22:05.404852 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.404803 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2vq\" (UniqueName: \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.405043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.404868 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.424402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.424330 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:22:05.428933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.428910 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.451332 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.451293 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:22:05.506739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.506690 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.506949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.506756 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2vq\" (UniqueName: \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.506949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.506840 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msq7j\" (UniqueName: \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.506949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.506884 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.507635 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.507612 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.526424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.526390 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2vq\" (UniqueName: \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\") pod \"payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.608302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.608199 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.608302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.608259 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msq7j\" (UniqueName: \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.609015 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.608987 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.613671 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.613643 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:05.628094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.628058 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msq7j\" (UniqueName: \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\") pod \"payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.743916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.743846 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:05.920405 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:22:05.920356 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e186910_179a_4b33_bc70_1787f40b66c3.slice/crio-6045eab4a4bd996519fc7c738e2b98521a0776ea046f4a03ae96154226a78b4c WatchSource:0}: Error finding container 6045eab4a4bd996519fc7c738e2b98521a0776ea046f4a03ae96154226a78b4c: Status 404 returned error can't find the container with id 6045eab4a4bd996519fc7c738e2b98521a0776ea046f4a03ae96154226a78b4c Oct 30 20:22:05.920573 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:05.920456 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:22:06.051586 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:22:06.051550 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0244bf_ad49_4a98_b3b0_e95f9c280961.slice/crio-5995b9dcc775303757200c511ae9ad5819b738b2c6bfea85b5d0b7dd5ab24099 WatchSource:0}: Error finding container 5995b9dcc775303757200c511ae9ad5819b738b2c6bfea85b5d0b7dd5ab24099: Status 404 returned error can't find the container with id 5995b9dcc775303757200c511ae9ad5819b738b2c6bfea85b5d0b7dd5ab24099 Oct 30 20:22:06.051726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.051610 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:22:06.322599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.322549 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" event={"ID":"5f0244bf-ad49-4a98-b3b0-e95f9c280961","Type":"ContainerStarted","Data":"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302"} Oct 30 20:22:06.323078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.322630 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:06.323078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.322680 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" event={"ID":"5f0244bf-ad49-4a98-b3b0-e95f9c280961","Type":"ContainerStarted","Data":"5995b9dcc775303757200c511ae9ad5819b738b2c6bfea85b5d0b7dd5ab24099"} Oct 30 20:22:06.324840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.324811 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" event={"ID":"8e186910-179a-4b33-bc70-1787f40b66c3","Type":"ContainerStarted","Data":"bef0df7fcbf3d1df8f0e40eaaffb4f1134ed1d81f9c756d7162f884be900a49e"} Oct 30 20:22:06.325003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.324858 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" event={"ID":"8e186910-179a-4b33-bc70-1787f40b66c3","Type":"ContainerStarted","Data":"6045eab4a4bd996519fc7c738e2b98521a0776ea046f4a03ae96154226a78b4c"} Oct 30 20:22:06.325003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.324920 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:06.365720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.365662 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" podStartSLOduration=1.365646316 podStartE2EDuration="1.365646316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:06.363764639 +0000 UTC m=+2793.813257363" watchObservedRunningTime="2026-10-30 20:22:06.365646316 +0000 UTC m=+2793.815139041" Oct 30 20:22:06.404204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:06.404139 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" podStartSLOduration=1.40412295 podStartE2EDuration="1.40412295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:06.402266734 +0000 UTC m=+2793.851759458" watchObservedRunningTime="2026-10-30 20:22:06.40412295 +0000 UTC m=+2793.853615675" Oct 30 20:22:17.336132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:17.336047 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:22:17.336132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:22:17.336121 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:22:20.870586 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:22:20.870545 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:22:50.874105 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:22:50.874060 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:23:02.823252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:02.823215 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:23:20.871122 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:23:20.871083 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:23:27.847743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:27.847709 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:23:27.852669 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:27.852647 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:27.870258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:27.870222 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:23:27.878425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:27.878397 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fbda3-dockercfg-82xbc\"" Oct 30 20:23:27.878425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:27.878413 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fbda3\"" Oct 30 20:23:28.036501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.036463 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxhn\" (UniqueName: \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.036724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.036628 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.038550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.038334 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:23:28.043719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.043697 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.049210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.049182 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:23:28.138095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.137988 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxhn\" (UniqueName: \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.138301 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.138145 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptfv\" (UniqueName: \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.138301 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.138185 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.138530 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.138501 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.138972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.138948 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.158255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.158220 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxhn\" (UniqueName: \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\") pod \"payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.167145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.167110 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.240416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.240341 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptfv\" (UniqueName: \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.240580 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.240546 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.241476 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.241451 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.261609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.261572 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptfv\" (UniqueName: \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.362745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.362711 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:28.486613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.486579 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:23:28.487529 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:23:28.487501 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5105e7d1_170e_4548_a210_3b53c12e2abf.slice/crio-73c79017639085bee5fbe800e176d8fbdbb672928cead60b3404bff3507fe50d WatchSource:0}: Error finding container 73c79017639085bee5fbe800e176d8fbdbb672928cead60b3404bff3507fe50d: Status 404 returned error can't find the container with id 73c79017639085bee5fbe800e176d8fbdbb672928cead60b3404bff3507fe50d Oct 30 20:23:28.667734 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:23:28.667697 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244ecbe2_ab8c_4c3f_ae48_57803ece337c.slice/crio-c3893005e54cc86a1f104ec854358a6a73ce2ee3112bc96142916ef756e7a5c1 WatchSource:0}: Error finding container c3893005e54cc86a1f104ec854358a6a73ce2ee3112bc96142916ef756e7a5c1: Status 404 returned error can't find the container with id c3893005e54cc86a1f104ec854358a6a73ce2ee3112bc96142916ef756e7a5c1 Oct 30 20:23:28.668941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.668918 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:23:28.966909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.966807 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" event={"ID":"5105e7d1-170e-4548-a210-3b53c12e2abf","Type":"ContainerStarted","Data":"b15cbd4589155dfa9fade002cc6cd5d86fd78264265fbd30996dc1c804648700"} Oct 30 20:23:28.966909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.966873 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" event={"ID":"5105e7d1-170e-4548-a210-3b53c12e2abf","Type":"ContainerStarted","Data":"73c79017639085bee5fbe800e176d8fbdbb672928cead60b3404bff3507fe50d"} Oct 30 20:23:28.966909 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.966908 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:28.969026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.968994 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" event={"ID":"244ecbe2-ab8c-4c3f-ae48-57803ece337c","Type":"ContainerStarted","Data":"2b46f5f0501a13b33f0e08a4fbb4f1b92099f8d6fb5736de91327aef66bb1099"} Oct 30 20:23:28.969175 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.969051 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" event={"ID":"244ecbe2-ab8c-4c3f-ae48-57803ece337c","Type":"ContainerStarted","Data":"c3893005e54cc86a1f104ec854358a6a73ce2ee3112bc96142916ef756e7a5c1"} Oct 30 20:23:28.969175 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:28.969082 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:29.010791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:29.010731 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" podStartSLOduration=2.010714501 podStartE2EDuration="2.010714501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:29.007839812 +0000 UTC m=+2876.457332537" watchObservedRunningTime="2026-10-30 20:23:29.010714501 +0000 UTC m=+2876.460207253" Oct 30 20:23:29.054752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:29.054534 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" podStartSLOduration=1.054515011 podStartE2EDuration="1.054515011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:29.049749985 +0000 UTC m=+2876.499242710" watchObservedRunningTime="2026-10-30 20:23:29.054515011 +0000 UTC m=+2876.504007737" Oct 30 20:23:37.620467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.620434 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:23:37.625703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.625674 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.640359 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.640336 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-043892-openshift-default-dockercfg-hvz52\"" Oct 30 20:23:37.657064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.657021 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:23:37.676857 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.676821 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.676883 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.676997 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677176 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677071 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677181 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr89n\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677291 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677384 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677344 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677476 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677461 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.677583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.677570 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.778751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.778714 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.778978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.778775 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.778978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.778829 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.778978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.778940 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779198 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779300 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779306 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779355 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779448 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779484 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779556 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779613 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr89n\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779781 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779718 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.779879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.779842 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.781332 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.781308 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.781869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.781846 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.799221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.799180 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.799421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.799326 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr89n\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-44w27\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:37.945080 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:37.944990 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:38.277473 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:38.275289 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:23:38.277473 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:23:38.277405 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f6375d_26ca_4195_91aa_56bfd1a3a4ab.slice/crio-b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147 WatchSource:0}: Error finding container b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147: Status 404 returned error can't find the container with id b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147 Oct 30 20:23:38.280235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:38.280194 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:38.280339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:38.280275 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:38.280339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:38.280307 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:39.064076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.064034 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" event={"ID":"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab","Type":"ContainerStarted","Data":"221bd53af5ba42dacdbbf24ff25feeb531c836b27988d7bc95607fa86c433a42"} Oct 30 20:23:39.064602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.064089 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" event={"ID":"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab","Type":"ContainerStarted","Data":"b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147"} Oct 30 20:23:39.121315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.121251 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podStartSLOduration=2.121236991 podStartE2EDuration="2.121236991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:39.118431872 +0000 UTC m=+2886.567924597" watchObservedRunningTime="2026-10-30 20:23:39.121236991 +0000 UTC m=+2886.570729718" Oct 30 20:23:39.945594 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.945549 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:39.947307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.947280 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:39.947400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.947334 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:39.990788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.990748 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:23:39.991213 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:39.991188 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:23:40.945612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:40.945571 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:40.946067 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:40.945637 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:41.946280 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:41.946243 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:41.946724 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:41.946304 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:42.945904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:42.945869 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:42.946106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:42.945925 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:43.946445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:43.946302 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:43.946445 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:43.946417 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:44.946114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:44.946071 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:23:44.946281 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:44.946143 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 20:23:45.951895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:45.951854 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:45.952398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:45.952340 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:45.953245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:23:45.953223 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:23:50.877278 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:23:50.877218 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:24:04.791340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:04.791297 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:24:19.817814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:19.817772 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" podUID="124bee40-600e-4e23-8487-fff9803301c0" containerName="main" containerID="cri-o://b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf" gracePeriod=45 Oct 30 20:24:20.130902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.130868 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:24:20.279159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279119 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.279347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279222 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\" (UniqueName: \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.279347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279316 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.279487 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279463 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:20.279625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279610 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.279691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279667 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.279828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.279809 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\") pod \"124bee40-600e-4e23-8487-fff9803301c0\" (UID: \"124bee40-600e-4e23-8487-fff9803301c0\") " Oct 30 20:24:20.280044 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.280014 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:20.280161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.280089 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:20.280520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.280504 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.280578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.280522 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.280578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.280534 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.281775 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.281749 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "kube-api-access-ff8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:20.281946 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.281931 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:20.282032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.282011 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm" pod "124bee40-600e-4e23-8487-fff9803301c0" (UID: "124bee40-600e-4e23-8487-fff9803301c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:20.381984 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.381900 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124bee40-600e-4e23-8487-fff9803301c0-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.381984 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.381932 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff8d8\" (UniqueName: \"kubernetes.io/projected/124bee40-600e-4e23-8487-fff9803301c0-kube-api-access-ff8d8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.381984 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.381946 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124bee40-600e-4e23-8487-fff9803301c0-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:20.434354 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.434321 5991 generic.go:356] "Generic (PLEG): container finished" podID="124bee40-600e-4e23-8487-fff9803301c0" containerID="b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf" exitCode=0 Oct 30 20:24:20.434354 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.434380 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" Oct 30 20:24:20.434625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.434402 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" event={"ID":"124bee40-600e-4e23-8487-fff9803301c0","Type":"ContainerDied","Data":"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf"} Oct 30 20:24:20.434625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.434449 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l" event={"ID":"124bee40-600e-4e23-8487-fff9803301c0","Type":"ContainerDied","Data":"42a9fb0b05e71fb90f433701a0512efc259e00b9059edd613374f6cc1c91d17f"} Oct 30 20:24:20.434625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.434466 5991 scope.go:122] "RemoveContainer" containerID="b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf" Oct 30 20:24:20.446737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.446707 5991 scope.go:122] "RemoveContainer" containerID="b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf" Oct 30 20:24:20.447105 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:24:20.447074 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf\": container with ID starting with b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf not found: ID does not exist" containerID="b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf" Oct 30 20:24:20.447176 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.447122 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf"} err="failed to get container status \"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf\": rpc error: code = NotFound desc = could not find container \"b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf\": container with ID starting with b2197c28ce0f2931f632308f464bd53717d1c2eb6aa6544cd5909504462e34cf not found: ID does not exist" Oct 30 20:24:20.487796 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.487759 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:24:20.491884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.491845 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-159b3a/sub-test-model-159b3a-kserve-76bbbf8bff-p5k2l"] Oct 30 20:24:20.681301 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:20.681216 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124bee40-600e-4e23-8487-fff9803301c0" path="/var/lib/kubelet/pods/124bee40-600e-4e23-8487-fff9803301c0/volumes" Oct 30 20:24:20.871577 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:24:20.871534 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:24:27.031630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.031585 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:24:27.033477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.033446 5991 cpu_manager.go:414] "containerMap: removing container" podUID="124bee40-600e-4e23-8487-fff9803301c0" containerName="main" Oct 30 20:24:27.033686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.033667 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124bee40-600e-4e23-8487-fff9803301c0" containerName="main" Oct 30 20:24:27.034055 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.034034 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="124bee40-600e-4e23-8487-fff9803301c0" containerName="main" Oct 30 20:24:27.061610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.052583 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:24:27.061610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.052734 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.087891 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.087821 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l925m\" (UniqueName: \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.088378 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.088303 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.110128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.110086 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c19563\"" Oct 30 20:24:27.110128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.110087 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c19563-dockercfg-wtxz2\"" Oct 30 20:24:27.189806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.189762 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l925m\" (UniqueName: \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.190014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.189948 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.190754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.190723 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.211441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.211404 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l925m\" (UniqueName: \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\") pod \"payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.392828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.392724 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:27.702242 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:24:27.702209 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe800b0d_8e38_4d92_bbf6_e67b9af49f89.slice/crio-2397d578fb737a55805306e24c153b9b04675308d204d3436fc2a23302bb3373 WatchSource:0}: Error finding container 2397d578fb737a55805306e24c153b9b04675308d204d3436fc2a23302bb3373: Status 404 returned error can't find the container with id 2397d578fb737a55805306e24c153b9b04675308d204d3436fc2a23302bb3373 Oct 30 20:24:27.702740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:27.702714 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:24:28.504114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:28.504076 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" event={"ID":"be800b0d-8e38-4d92-bbf6-e67b9af49f89","Type":"ContainerStarted","Data":"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8"} Oct 30 20:24:28.504568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:28.504130 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" event={"ID":"be800b0d-8e38-4d92-bbf6-e67b9af49f89","Type":"ContainerStarted","Data":"2397d578fb737a55805306e24c153b9b04675308d204d3436fc2a23302bb3373"} Oct 30 20:24:28.504568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:28.504282 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:28.561938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:28.561879 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" podStartSLOduration=2.561862482 podStartE2EDuration="2.561862482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:28.557756504 +0000 UTC m=+2936.007249230" watchObservedRunningTime="2026-10-30 20:24:28.561862482 +0000 UTC m=+2936.011355242" Oct 30 20:24:31.125389 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.123924 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:24:31.130838 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.130806 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.160446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.160408 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:24:31.165598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.165563 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7eec8\"" Oct 30 20:24:31.251163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.251125 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkb4\" (UniqueName: \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.251425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.251278 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.352962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.352922 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkb4\" (UniqueName: \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.353197 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.353000 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.353804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.353782 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.373978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.373945 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkb4\" (UniqueName: \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.449736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.449652 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:31.758639 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:24:31.758595 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5aef768_0b9a_43fa_aeb8_a45744dc2830.slice/crio-5b1cebd474f53a184abb0d70a2a4bbf75d7a7378b9258f19730f5fa2ad43506e WatchSource:0}: Error finding container 5b1cebd474f53a184abb0d70a2a4bbf75d7a7378b9258f19730f5fa2ad43506e: Status 404 returned error can't find the container with id 5b1cebd474f53a184abb0d70a2a4bbf75d7a7378b9258f19730f5fa2ad43506e Oct 30 20:24:31.758797 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:31.758720 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:24:32.564762 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:32.564723 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" event={"ID":"a5aef768-0b9a-43fa-aeb8-a45744dc2830","Type":"ContainerStarted","Data":"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173"} Oct 30 20:24:32.565245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:32.564803 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" event={"ID":"a5aef768-0b9a-43fa-aeb8-a45744dc2830","Type":"ContainerStarted","Data":"5b1cebd474f53a184abb0d70a2a4bbf75d7a7378b9258f19730f5fa2ad43506e"} Oct 30 20:24:32.565482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:32.565438 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:32.608637 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:32.608579 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" podStartSLOduration=1.608561073 podStartE2EDuration="1.608561073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:32.605864994 +0000 UTC m=+2940.055357719" watchObservedRunningTime="2026-10-30 20:24:32.608561073 +0000 UTC m=+2940.058053823" Oct 30 20:24:39.516528 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:39.516497 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:24:42.390493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:42.390454 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:24:42.391010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:42.390818 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" containerID="cri-o://221bd53af5ba42dacdbbf24ff25feeb531c836b27988d7bc95607fa86c433a42" gracePeriod=30 Oct 30 20:24:43.575718 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:43.575687 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:24:45.954859 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:45.954815 5991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:24:45.955283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:45.954895 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:24:47.689578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.689538 5991 generic.go:356] "Generic (PLEG): container finished" podID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerID="221bd53af5ba42dacdbbf24ff25feeb531c836b27988d7bc95607fa86c433a42" exitCode=0 Oct 30 20:24:47.690004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.689607 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" event={"ID":"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab","Type":"ContainerDied","Data":"221bd53af5ba42dacdbbf24ff25feeb531c836b27988d7bc95607fa86c433a42"} Oct 30 20:24:47.690004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.689677 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" event={"ID":"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab","Type":"ContainerDied","Data":"b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147"} Oct 30 20:24:47.690004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.689705 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7f8d9de78f546e6245b9f686c5557ecae50dceaf12905c30e0664b1c441c147" Oct 30 20:24:47.698392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.698352 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:24:47.808443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808399 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808460 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808519 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808552 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808604 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808661 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808730 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.808899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808739 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.808899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808849 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.809123 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.808937 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.809123 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809008 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\") pod \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\" (UID: \"d3f6375d-26ca-4195-91aa-56bfd1a3a4ab\") " Oct 30 20:24:47.809244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809145 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.809514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809488 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.809765 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809735 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:47.809894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809772 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.809894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809793 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.809894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809808 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.809894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.809821 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.811882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.811844 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:47.811882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.811854 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.812312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.812281 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:47.812631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.812605 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n" pod "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" (UID: "d3f6375d-26ca-4195-91aa-56bfd1a3a4ab"). InnerVolumeSpecName "kube-api-access-wr89n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:47.911125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.911036 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.911125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.911067 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.911125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.911079 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr89n\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-kube-api-access-wr89n\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.911125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.911090 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.911125 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:47.911104 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:48.700758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:48.700698 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27" Oct 30 20:24:48.773982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:48.773945 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:24:48.780965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:48.780927 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-44w27"] Oct 30 20:24:50.684426 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:50.684382 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" path="/var/lib/kubelet/pods/d3f6375d-26ca-4195-91aa-56bfd1a3a4ab/volumes" Oct 30 20:24:50.874783 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:24:50.874738 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:24:55.106636 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:55.106600 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:24:57.245524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.245471 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:24:57.246099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.245762 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" podUID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" containerName="payload-pre-processing" containerID="cri-o://82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b" gracePeriod=30 Oct 30 20:24:57.614674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.614254 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:24:57.653698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.653624 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\") pod \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " Oct 30 20:24:57.653926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.653782 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\" (UniqueName: \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\") pod \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\" (UID: \"9dff2bea-e37c-450e-98ad-109c2b70f6f3\") " Oct 30 20:24:57.660243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.660205 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh" pod "9dff2bea-e37c-450e-98ad-109c2b70f6f3" (UID: "9dff2bea-e37c-450e-98ad-109c2b70f6f3"). InnerVolumeSpecName "kube-api-access-bltlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:57.660590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.660563 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume" pod "9dff2bea-e37c-450e-98ad-109c2b70f6f3" (UID: "9dff2bea-e37c-450e-98ad-109c2b70f6f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:57.756819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.756779 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dff2bea-e37c-450e-98ad-109c2b70f6f3-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.756819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.756810 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltlh\" (UniqueName: \"kubernetes.io/projected/9dff2bea-e37c-450e-98ad-109c2b70f6f3-kube-api-access-bltlh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.817130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.817034 5991 generic.go:356] "Generic (PLEG): container finished" podID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" containerID="82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b" exitCode=0 Oct 30 20:24:57.817130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.817081 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" Oct 30 20:24:57.817130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.817116 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" event={"ID":"9dff2bea-e37c-450e-98ad-109c2b70f6f3","Type":"ContainerDied","Data":"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b"} Oct 30 20:24:57.817481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.817173 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p" event={"ID":"9dff2bea-e37c-450e-98ad-109c2b70f6f3","Type":"ContainerDied","Data":"5eca8c52f5b54fa473e114222f04ec2497b6f00f7ae13095b889871054d0c9cf"} Oct 30 20:24:57.817481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.817276 5991 scope.go:122] "RemoveContainer" containerID="82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b" Oct 30 20:24:57.828672 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.828651 5991 scope.go:122] "RemoveContainer" containerID="82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b" Oct 30 20:24:57.829000 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:24:57.828978 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b\": container with ID starting with 82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b not found: ID does not exist" containerID="82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b" Oct 30 20:24:57.829048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.829010 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b"} err="failed to get container status \"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b\": rpc error: code = NotFound desc = could not find container \"82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b\": container with ID starting with 82d515db2d88b2651c8a72606781a1ff2f019b24cf30ba7cc9388cd0c1690a0b not found: ID does not exist" Oct 30 20:24:57.873229 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.873190 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:24:57.876906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:57.876878 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-6f8b7c979f-m4j7p"] Oct 30 20:24:58.683240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:24:58.683184 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" path="/var/lib/kubelet/pods/9dff2bea-e37c-450e-98ad-109c2b70f6f3/volumes" Oct 30 20:25:06.063333 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:06.063291 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:25:06.063837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:06.063663 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" containerID="cri-o://3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c" gracePeriod=30 Oct 30 20:25:06.320751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:06.320668 5991 patch_prober.go:29] interesting pod/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:25:06.320751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:06.320744 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:25:08.112509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:08.112462 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:25:08.113019 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:08.112735 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" containerID="cri-o://eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924" gracePeriod=30 Oct 30 20:25:11.122645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.122612 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 20:25:11.135739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.135690 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 20:25:11.387154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.387116 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:25:11.475777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.475737 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.475841 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.475906 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.475939 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476029 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476029 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.476219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476166 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476225 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476260 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476271 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476395 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\") pod \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\" (UID: \"fdfb0705-0c49-4e44-a2af-f3c042ad6662\") " Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476468 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.476509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476358 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:11.476836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.476590 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.477266 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.477241 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.477515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.477498 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.477637 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.477622 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.477734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.477722 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.477828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.477817 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.479299 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.479270 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:11.479472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.479394 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:11.479472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.479436 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.480542 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.480517 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg" pod "fdfb0705-0c49-4e44-a2af-f3c042ad6662" (UID: "fdfb0705-0c49-4e44-a2af-f3c042ad6662"). InnerVolumeSpecName "kube-api-access-6qjbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:11.579147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.579113 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.579147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.579143 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjbg\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-kube-api-access-6qjbg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.579147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.579155 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.579431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.579166 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfb0705-0c49-4e44-a2af-f3c042ad6662-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.927889 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.927850 5991 generic.go:356] "Generic (PLEG): container finished" podID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerID="3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c" exitCode=0 Oct 30 20:25:11.928092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.927930 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" Oct 30 20:25:11.928092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.927939 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerDied","Data":"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c"} Oct 30 20:25:11.928092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.927999 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q" event={"ID":"fdfb0705-0c49-4e44-a2af-f3c042ad6662","Type":"ContainerDied","Data":"d9972c206bbd093244f252dcb205d865eb161a2ab36beed2819297bcac70184c"} Oct 30 20:25:11.928092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.928023 5991 scope.go:122] "RemoveContainer" containerID="3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c" Oct 30 20:25:11.940661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.940635 5991 scope.go:122] "RemoveContainer" containerID="1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e" Oct 30 20:25:11.952246 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.952220 5991 scope.go:122] "RemoveContainer" containerID="3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c" Oct 30 20:25:11.952629 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:11.952608 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c\": container with ID starting with 3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c not found: ID does not exist" containerID="3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c" Oct 30 20:25:11.952698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.952639 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c"} err="failed to get container status \"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c\": rpc error: code = NotFound desc = could not find container \"3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c\": container with ID starting with 3a838db2937d872fd11a7af354d3dfac00036285154a8f0e817745defc6c416c not found: ID does not exist" Oct 30 20:25:11.952698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.952658 5991 scope.go:122] "RemoveContainer" containerID="1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e" Oct 30 20:25:11.952948 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:11.952927 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e\": container with ID starting with 1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e not found: ID does not exist" containerID="1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e" Oct 30 20:25:11.952988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.952954 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e"} err="failed to get container status \"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e\": rpc error: code = NotFound desc = could not find container \"1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e\": container with ID starting with 1521bc96c4b81e0dd75b42de84898963baa858f1bfc62efe267ca0e9f17d154e not found: ID does not exist" Oct 30 20:25:11.985703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.985666 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:25:11.997975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:11.997940 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6175d5-openshift-default-66fbdb9f4c-bs78q"] Oct 30 20:25:12.680536 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:12.680491 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" path="/var/lib/kubelet/pods/fdfb0705-0c49-4e44-a2af-f3c042ad6662/volumes" Oct 30 20:25:13.397234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.397161 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:25:13.505377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505327 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505393 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505437 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505529 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505696 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505805 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.505998 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.505971 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:13.506102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506039 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:13.506187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506167 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:13.506288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506270 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.506417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506311 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.506417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506407 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\") pod \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\" (UID: \"605ab214-f577-4cd3-aa4b-2794c2c8f21d\") " Oct 30 20:25:13.506554 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506274 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:13.507025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.506900 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:13.507541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.507453 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.507541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.507479 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.507541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.507492 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.507541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.507505 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.507541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.507518 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.508882 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.508858 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "kube-api-access-cs4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:13.508976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.508858 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:13.508976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.508908 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:13.509083 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.509030 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy" pod "605ab214-f577-4cd3-aa4b-2794c2c8f21d" (UID: "605ab214-f577-4cd3-aa4b-2794c2c8f21d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:13.609041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.609002 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4bh\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-kube-api-access-cs4bh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.609041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.609038 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.609041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.609048 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.609311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.609056 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605ab214-f577-4cd3-aa4b-2794c2c8f21d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:13.945311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.945278 5991 generic.go:356] "Generic (PLEG): container finished" podID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerID="eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924" exitCode=0 Oct 30 20:25:13.945795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.945326 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" Oct 30 20:25:13.945795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.945356 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" event={"ID":"605ab214-f577-4cd3-aa4b-2794c2c8f21d","Type":"ContainerDied","Data":"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924"} Oct 30 20:25:13.945795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.945429 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s" event={"ID":"605ab214-f577-4cd3-aa4b-2794c2c8f21d","Type":"ContainerDied","Data":"8ed9947eb013a6b9af1e2ab21e588ef3dea34333662493d43dc93fbb8c74e27d"} Oct 30 20:25:13.945795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.945454 5991 scope.go:122] "RemoveContainer" containerID="eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924" Oct 30 20:25:13.957630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.957609 5991 scope.go:122] "RemoveContainer" containerID="eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924" Oct 30 20:25:13.957978 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:13.957938 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924\": container with ID starting with eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924 not found: ID does not exist" containerID="eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924" Oct 30 20:25:13.958028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:13.957986 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924"} err="failed to get container status \"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924\": rpc error: code = NotFound desc = could not find container \"eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924\": container with ID starting with eb6f7b165e4bab672923cc3c6085fbc62970374820dee49262abdd5733d29924 not found: ID does not exist" Oct 30 20:25:14.012316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:14.012272 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:25:14.024795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:14.024757 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-8fw4s"] Oct 30 20:25:14.681503 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:14.681456 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" path="/var/lib/kubelet/pods/605ab214-f577-4cd3-aa4b-2794c2c8f21d/volumes" Oct 30 20:25:17.137205 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:17.137167 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 20:25:20.871504 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:20.871469 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:25:20.954845 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:20.954803 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:25:20.955084 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:20.955031 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" podUID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" containerName="payload-pre-processing" containerID="cri-o://02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173" gracePeriod=30 Oct 30 20:25:21.257287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.257247 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:25:21.419735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.419637 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\" (UniqueName: \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\") pod \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " Oct 30 20:25:21.419735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.419715 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\") pod \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\" (UID: \"a5aef768-0b9a-43fa-aeb8-a45744dc2830\") " Oct 30 20:25:21.420171 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.420140 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume" pod "a5aef768-0b9a-43fa-aeb8-a45744dc2830" (UID: "a5aef768-0b9a-43fa-aeb8-a45744dc2830"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:21.422028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.422005 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4" pod "a5aef768-0b9a-43fa-aeb8-a45744dc2830" (UID: "a5aef768-0b9a-43fa-aeb8-a45744dc2830"). InnerVolumeSpecName "kube-api-access-czkb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:21.521832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.521794 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkb4\" (UniqueName: \"kubernetes.io/projected/a5aef768-0b9a-43fa-aeb8-a45744dc2830-kube-api-access-czkb4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:21.521832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:21.521829 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aef768-0b9a-43fa-aeb8-a45744dc2830-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:22.011268 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.011229 5991 generic.go:356] "Generic (PLEG): container finished" podID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" containerID="02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173" exitCode=0 Oct 30 20:25:22.011833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.011280 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" Oct 30 20:25:22.011833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.011320 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" event={"ID":"a5aef768-0b9a-43fa-aeb8-a45744dc2830","Type":"ContainerDied","Data":"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173"} Oct 30 20:25:22.011833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.011411 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm" event={"ID":"a5aef768-0b9a-43fa-aeb8-a45744dc2830","Type":"ContainerDied","Data":"5b1cebd474f53a184abb0d70a2a4bbf75d7a7378b9258f19730f5fa2ad43506e"} Oct 30 20:25:22.011833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.011444 5991 scope.go:122] "RemoveContainer" containerID="02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173" Oct 30 20:25:22.024221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.024193 5991 scope.go:122] "RemoveContainer" containerID="02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173" Oct 30 20:25:22.024625 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:22.024601 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173\": container with ID starting with 02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173 not found: ID does not exist" containerID="02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173" Oct 30 20:25:22.024721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.024634 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173"} err="failed to get container status \"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173\": rpc error: code = NotFound desc = could not find container \"02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173\": container with ID starting with 02278dd9bebf7d57c9df4b79dd116c0cf10062cd3cb963316c540a07c6085173 not found: ID does not exist" Oct 30 20:25:22.070473 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.070435 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:25:22.082234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.082199 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-74f657f467-jghcm"] Oct 30 20:25:22.138452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.138354 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" containerID="cri-o://b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8" gracePeriod=25 Oct 30 20:25:22.681687 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:22.681651 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" path="/var/lib/kubelet/pods/a5aef768-0b9a-43fa-aeb8-a45744dc2830/volumes" Oct 30 20:25:26.147323 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.147279 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" containerID="cri-o://ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1" gracePeriod=45 Oct 30 20:25:26.162941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.162865 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" containerID="cri-o://5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc" gracePeriod=45 Oct 30 20:25:26.231132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.231090 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused" Oct 30 20:25:26.477103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.477078 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 20:25:26.480839 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.480813 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 20:25:26.492279 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492247 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492490 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492407 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492490 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492443 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492529 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492589 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\" (UniqueName: \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492582 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.492791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492672 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.492791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492723 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\") pod \"42e12231-4859-49d4-9c38-d485230c0a6b\" (UID: \"42e12231-4859-49d4-9c38-d485230c0a6b\") " Oct 30 20:25:26.492907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.492846 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.493358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.493333 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.493489 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.493393 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.493489 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.493407 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.495060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.495030 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.495166 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.495054 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:26.495166 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.495074 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj" pod "42e12231-4859-49d4-9c38-d485230c0a6b" (UID: "42e12231-4859-49d4-9c38-d485230c0a6b"). InnerVolumeSpecName "kube-api-access-7gncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.596830 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.596920 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.596974 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.597060 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.597244 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.597359 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\") pod \"b42b2133-af7b-4713-966b-e98f5e14e3f4\" (UID: \"b42b2133-af7b-4713-966b-e98f5e14e3f4\") " Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.598011 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e12231-4859-49d4-9c38-d485230c0a6b-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.598031 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e12231-4859-49d4-9c38-d485230c0a6b-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.598044 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gncj\" (UniqueName: \"kubernetes.io/projected/42e12231-4859-49d4-9c38-d485230c0a6b-kube-api-access-7gncj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.598582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.598293 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.599320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.599074 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.599320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.599265 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.601320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.601288 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "kube-api-access-qcw82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:26.602271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.602243 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.602700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.602677 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs" pod "b42b2133-af7b-4713-966b-e98f5e14e3f4" (UID: "b42b2133-af7b-4713-966b-e98f5e14e3f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:26.699345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699320 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2133-af7b-4713-966b-e98f5e14e3f4-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.699345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699344 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.699582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699388 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.699582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699420 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.699582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699436 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/b42b2133-af7b-4713-966b-e98f5e14e3f4-kube-api-access-qcw82\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.699582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:26.699446 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2133-af7b-4713-966b-e98f5e14e3f4-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:27.053943 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.053908 5991 generic.go:356] "Generic (PLEG): container finished" podID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerID="5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc" exitCode=0 Oct 30 20:25:27.054165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.053953 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" Oct 30 20:25:27.054165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.053994 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" event={"ID":"b42b2133-af7b-4713-966b-e98f5e14e3f4","Type":"ContainerDied","Data":"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc"} Oct 30 20:25:27.054165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.054036 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n" event={"ID":"b42b2133-af7b-4713-966b-e98f5e14e3f4","Type":"ContainerDied","Data":"e66f2318ad737f1657e2cda268fac8a268a680b704266570915eb746ee3537c6"} Oct 30 20:25:27.054165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.054057 5991 scope.go:122] "RemoveContainer" containerID="5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc" Oct 30 20:25:27.055692 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.055671 5991 generic.go:356] "Generic (PLEG): container finished" podID="42e12231-4859-49d4-9c38-d485230c0a6b" containerID="ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1" exitCode=0 Oct 30 20:25:27.055803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.055723 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" Oct 30 20:25:27.055803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.055763 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" event={"ID":"42e12231-4859-49d4-9c38-d485230c0a6b","Type":"ContainerDied","Data":"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1"} Oct 30 20:25:27.055803 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.055797 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp" event={"ID":"42e12231-4859-49d4-9c38-d485230c0a6b","Type":"ContainerDied","Data":"a2bcda5cf41b86be8e776c7f5811707f9e28ecfbbc0c2216193f2389b19d0b4b"} Oct 30 20:25:27.065996 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.065975 5991 scope.go:122] "RemoveContainer" containerID="5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc" Oct 30 20:25:27.066331 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:27.066310 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc\": container with ID starting with 5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc not found: ID does not exist" containerID="5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc" Oct 30 20:25:27.066405 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.066342 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc"} err="failed to get container status \"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc\": rpc error: code = NotFound desc = could not find container \"5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc\": container with ID starting with 5e3a38b668458629cd9ac1e41bb8bac27223ce6d01ed9b7a4f26b7e52a20abbc not found: ID does not exist" Oct 30 20:25:27.066405 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.066390 5991 scope.go:122] "RemoveContainer" containerID="ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1" Oct 30 20:25:27.076149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.076122 5991 scope.go:122] "RemoveContainer" containerID="ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1" Oct 30 20:25:27.076495 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:27.076473 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1\": container with ID starting with ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1 not found: ID does not exist" containerID="ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1" Oct 30 20:25:27.076584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.076503 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1"} err="failed to get container status \"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1\": rpc error: code = NotFound desc = could not find container \"ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1\": container with ID starting with ead3653e53b206ef89d9196757e7ec5fb4023faab12d7c9dc2077e151796abf1 not found: ID does not exist" Oct 30 20:25:27.108402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.108341 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 20:25:27.120417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.120385 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/premium-w4-8a3426-kserve-64d764f4bb-wsb7n"] Oct 30 20:25:27.147181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.147143 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 20:25:27.150592 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:27.150555 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-w4-8a3426-kserve-8957c8df7-mf9fp"] Oct 30 20:25:28.680103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:28.680068 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" path="/var/lib/kubelet/pods/42e12231-4859-49d4-9c38-d485230c0a6b/volumes" Oct 30 20:25:28.680585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:28.680570 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" path="/var/lib/kubelet/pods/b42b2133-af7b-4713-966b-e98f5e14e3f4/volumes" Oct 30 20:25:30.741918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:30.741881 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:33.247855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:33.247828 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:25:33.259510 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:33.259482 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:25:34.701152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:34.701119 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 20:25:34.713011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:34.712977 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 20:25:40.595844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:40.595804 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 20:25:40.741129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:40.741090 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:42.422084 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.422055 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 20:25:42.431181 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.431150 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\" (UniqueName: \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\") pod \"66499969-3c1f-4a18-816c-a30d6e470abb\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " Oct 30 20:25:42.431340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.431230 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\") pod \"66499969-3c1f-4a18-816c-a30d6e470abb\" (UID: \"66499969-3c1f-4a18-816c-a30d6e470abb\") " Oct 30 20:25:42.431614 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.431589 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume" pod "66499969-3c1f-4a18-816c-a30d6e470abb" (UID: "66499969-3c1f-4a18-816c-a30d6e470abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:42.431840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.431826 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66499969-3c1f-4a18-816c-a30d6e470abb-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:42.433417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.433395 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf" pod "66499969-3c1f-4a18-816c-a30d6e470abb" (UID: "66499969-3c1f-4a18-816c-a30d6e470abb"). InnerVolumeSpecName "kube-api-access-vmvtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:42.533551 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:42.533509 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmvtf\" (UniqueName: \"kubernetes.io/projected/66499969-3c1f-4a18-816c-a30d6e470abb-kube-api-access-vmvtf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:43.192154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.192118 5991 generic.go:356] "Generic (PLEG): container finished" podID="66499969-3c1f-4a18-816c-a30d6e470abb" containerID="b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8" exitCode=0 Oct 30 20:25:43.192344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.192182 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" Oct 30 20:25:43.192344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.192186 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" event={"ID":"66499969-3c1f-4a18-816c-a30d6e470abb","Type":"ContainerDied","Data":"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8"} Oct 30 20:25:43.192344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.192309 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr" event={"ID":"66499969-3c1f-4a18-816c-a30d6e470abb","Type":"ContainerDied","Data":"4e187072df2808e07f51bb42a745b7d272fd7b94b7dae80d5afd4299ace81f42"} Oct 30 20:25:43.192344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.192331 5991 scope.go:122] "RemoveContainer" containerID="b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8" Oct 30 20:25:43.204116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.204094 5991 scope.go:122] "RemoveContainer" containerID="b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8" Oct 30 20:25:43.204446 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:43.204417 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8\": container with ID starting with b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8 not found: ID does not exist" containerID="b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8" Oct 30 20:25:43.204566 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.204451 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8"} err="failed to get container status \"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8\": rpc error: code = NotFound desc = could not find container \"b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8\": container with ID starting with b535c96c88dc704f194b84290c0dfcdc5f5f4ceaef103c31a2550920f04aaca8 not found: ID does not exist" Oct 30 20:25:43.246153 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.246122 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 20:25:43.258468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:43.258435 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a3426-67bcc84c46-kr8tr"] Oct 30 20:25:44.680735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:44.680697 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" path="/var/lib/kubelet/pods/66499969-3c1f-4a18-816c-a30d6e470abb/volumes" Oct 30 20:25:45.596826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.596780 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" podUID="7bae6318-f6d7-4874-a331-784d18707f48" containerName="payload-pre-processing" containerID="cri-o://54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961" gracePeriod=25 Oct 30 20:25:45.875480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.875451 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 20:25:45.988823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.988785 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\" (UniqueName: \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\") pod \"7bae6318-f6d7-4874-a331-784d18707f48\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " Oct 30 20:25:45.988823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.988829 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\") pod \"7bae6318-f6d7-4874-a331-784d18707f48\" (UID: \"7bae6318-f6d7-4874-a331-784d18707f48\") " Oct 30 20:25:45.989289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.989244 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume" pod "7bae6318-f6d7-4874-a331-784d18707f48" (UID: "7bae6318-f6d7-4874-a331-784d18707f48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:45.991054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:45.991024 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr" pod "7bae6318-f6d7-4874-a331-784d18707f48" (UID: "7bae6318-f6d7-4874-a331-784d18707f48"). InnerVolumeSpecName "kube-api-access-b8ckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:46.091188 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.091148 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ckr\" (UniqueName: \"kubernetes.io/projected/7bae6318-f6d7-4874-a331-784d18707f48-kube-api-access-b8ckr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:46.091188 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.091186 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bae6318-f6d7-4874-a331-784d18707f48-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:46.219150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.219063 5991 generic.go:356] "Generic (PLEG): container finished" podID="7bae6318-f6d7-4874-a331-784d18707f48" containerID="54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961" exitCode=0 Oct 30 20:25:46.219150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.219115 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" Oct 30 20:25:46.219373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.219146 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" event={"ID":"7bae6318-f6d7-4874-a331-784d18707f48","Type":"ContainerDied","Data":"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961"} Oct 30 20:25:46.219373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.219203 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt" event={"ID":"7bae6318-f6d7-4874-a331-784d18707f48","Type":"ContainerDied","Data":"0137578ebf449ab5b1197d4bed90200cc87a3eb0e908fb8a2b477200e96825c5"} Oct 30 20:25:46.219373 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.219227 5991 scope.go:122] "RemoveContainer" containerID="54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961" Oct 30 20:25:46.230340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.230318 5991 scope.go:122] "RemoveContainer" containerID="54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961" Oct 30 20:25:46.230658 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:46.230641 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961\": container with ID starting with 54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961 not found: ID does not exist" containerID="54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961" Oct 30 20:25:46.230701 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.230667 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961"} err="failed to get container status \"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961\": rpc error: code = NotFound desc = could not find container \"54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961\": container with ID starting with 54390d00d99c1747d07dc3bd9737353c081a133a020a2721776d93704d44b961 not found: ID does not exist" Oct 30 20:25:46.272863 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.272830 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 20:25:46.284898 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.284864 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6c4dd1-5bb989c89f-tzkjt"] Oct 30 20:25:46.680586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:46.680551 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bae6318-f6d7-4874-a331-784d18707f48" path="/var/lib/kubelet/pods/7bae6318-f6d7-4874-a331-784d18707f48/volumes" Oct 30 20:25:47.576333 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:47.576290 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:25:49.724305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:49.724267 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" containerID="cri-o://479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963" gracePeriod=45 Oct 30 20:25:49.735569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:49.735534 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" containerID="cri-o://ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734" gracePeriod=45 Oct 30 20:25:49.864865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:49.864831 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:25:50.058505 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.058475 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 20:25:50.062103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.062084 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 20:25:50.151795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.151754 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.151980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.151840 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\" (UniqueName: \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.151980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.151931 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.152146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152038 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.152146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152123 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.152263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152187 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.152263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152215 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\" (UniqueName: \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.152263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152254 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.152467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152297 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.152467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152336 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\") pod \"af8af32d-b630-45ce-93e9-a1df20cff8b3\" (UID: \"af8af32d-b630-45ce-93e9-a1df20cff8b3\") " Oct 30 20:25:50.152467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152409 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.152467 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152437 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\") pod \"68bac318-6c9d-4df0-97fa-8a123eddda8c\" (UID: \"68bac318-6c9d-4df0-97fa-8a123eddda8c\") " Oct 30 20:25:50.152795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.152765 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.153240 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.153213 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.154095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.153822 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.154512 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.154268 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.154837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.154646 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.155020 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.154977 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.155196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.155154 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.155915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.155886 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:50.156421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.156240 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "kube-api-access-whbfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:50.156421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.156353 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "kube-api-access-jqqmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:50.156680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.156652 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:50.157320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.157294 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm" pod "68bac318-6c9d-4df0-97fa-8a123eddda8c" (UID: "68bac318-6c9d-4df0-97fa-8a123eddda8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.157589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.157558 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm" pod "af8af32d-b630-45ce-93e9-a1df20cff8b3" (UID: "af8af32d-b630-45ce-93e9-a1df20cff8b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:50.253341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.253244 5991 generic.go:356] "Generic (PLEG): container finished" podID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerID="ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734" exitCode=0 Oct 30 20:25:50.253524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.253452 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" Oct 30 20:25:50.254398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.254351 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" event={"ID":"68bac318-6c9d-4df0-97fa-8a123eddda8c","Type":"ContainerDied","Data":"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734"} Oct 30 20:25:50.254653 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.254413 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp" event={"ID":"68bac318-6c9d-4df0-97fa-8a123eddda8c","Type":"ContainerDied","Data":"901dddfc6d95016dd5031dd93e7aa322a5a5f9e30584e7b6d931e5ace0c6ad6e"} Oct 30 20:25:50.254653 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.254430 5991 scope.go:122] "RemoveContainer" containerID="ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734" Oct 30 20:25:50.255850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255820 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqqmq\" (UniqueName: \"kubernetes.io/projected/af8af32d-b630-45ce-93e9-a1df20cff8b3-kube-api-access-jqqmq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.255850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255845 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255858 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255867 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255876 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbfc\" (UniqueName: \"kubernetes.io/projected/68bac318-6c9d-4df0-97fa-8a123eddda8c-kube-api-access-whbfc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255883 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255893 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8af32d-b630-45ce-93e9-a1df20cff8b3-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255900 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af8af32d-b630-45ce-93e9-a1df20cff8b3-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255912 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255922 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bac318-6c9d-4df0-97fa-8a123eddda8c-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.256007 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.255929 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bac318-6c9d-4df0-97fa-8a123eddda8c-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:25:50.257207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.257179 5991 generic.go:356] "Generic (PLEG): container finished" podID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerID="479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963" exitCode=0 Oct 30 20:25:50.257478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.257446 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" event={"ID":"af8af32d-b630-45ce-93e9-a1df20cff8b3","Type":"ContainerDied","Data":"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963"} Oct 30 20:25:50.257575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.257489 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" event={"ID":"af8af32d-b630-45ce-93e9-a1df20cff8b3","Type":"ContainerDied","Data":"71aa6825d4352cf1c1f91e039256086fad8b7e56d9b96c913a7b2f24d2a53d65"} Oct 30 20:25:50.257575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.257562 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl" Oct 30 20:25:50.270423 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.270398 5991 scope.go:122] "RemoveContainer" containerID="ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734" Oct 30 20:25:50.270788 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:50.270763 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734\": container with ID starting with ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734 not found: ID does not exist" containerID="ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734" Oct 30 20:25:50.270892 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.270797 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734"} err="failed to get container status \"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734\": rpc error: code = NotFound desc = could not find container \"ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734\": container with ID starting with ff55f9a2a04ecfdc9e706a92751f82eab6256957b99cc25d3bfd69008b725734 not found: ID does not exist" Oct 30 20:25:50.270892 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.270816 5991 scope.go:122] "RemoveContainer" containerID="479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963" Oct 30 20:25:50.281431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.281403 5991 scope.go:122] "RemoveContainer" containerID="479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963" Oct 30 20:25:50.281823 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:50.281798 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963\": container with ID starting with 479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963 not found: ID does not exist" containerID="479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963" Oct 30 20:25:50.281980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.281834 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963"} err="failed to get container status \"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963\": rpc error: code = NotFound desc = could not find container \"479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963\": container with ID starting with 479325fa50f078ec24ad355103646bcadec2990baab76852e5b4e583a5651963 not found: ID does not exist" Oct 30 20:25:50.321402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.321345 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 20:25:50.324493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.324465 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-2-w5-6c4dd1-kserve-6cfdfb5c9-kxfbl"] Oct 30 20:25:50.372044 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.372010 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 20:25:50.375711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.375676 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/distinct-w5-6c4dd1-kserve-7bd4b976bd-wlfpp"] Oct 30 20:25:50.682417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.682314 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" path="/var/lib/kubelet/pods/68bac318-6c9d-4df0-97fa-8a123eddda8c/volumes" Oct 30 20:25:50.682978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:25:50.682953 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" path="/var/lib/kubelet/pods/af8af32d-b630-45ce-93e9-a1df20cff8b3/volumes" Oct 30 20:25:50.870654 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:25:50.870618 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:26:00.217965 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:00.217933 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:26:00.415806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:00.415775 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:26:00.608740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:00.608711 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv_5f0244bf-ad49-4a98-b3b0-e95f9c280961/payload-processing/0.log" Oct 30 20:26:00.817502 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:00.817476 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv_5f0244bf-ad49-4a98-b3b0-e95f9c280961/payload-processing/0.log" Oct 30 20:26:02.599359 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.599294 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" podUID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" containerName="main" containerID="cri-o://98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9" gracePeriod=45 Oct 30 20:26:02.907896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.907865 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:26:02.967029 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.966989 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\" (UniqueName: \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967135 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967187 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967242 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\" (UniqueName: \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967276 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967509 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\") pod \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\" (UID: \"915aea68-342e-4af2-9ea6-2f5d1ed3b84e\") " Oct 30 20:26:02.967781 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967754 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:02.967862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.967822 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:02.968263 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.968239 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:02.968386 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.968268 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:02.968461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.968447 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:02.970096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.969908 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "kube-api-access-nj278". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:02.970096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.970004 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:02.970096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:02.970010 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm" pod "915aea68-342e-4af2-9ea6-2f5d1ed3b84e" (UID: "915aea68-342e-4af2-9ea6-2f5d1ed3b84e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:03.069857 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.069794 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:03.070052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.069872 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj278\" (UniqueName: \"kubernetes.io/projected/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-kube-api-access-nj278\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:03.070052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.069893 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:03.070052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.069907 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915aea68-342e-4af2-9ea6-2f5d1ed3b84e-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:03.359282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.359246 5991 generic.go:356] "Generic (PLEG): container finished" podID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" containerID="98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9" exitCode=0 Oct 30 20:26:03.359524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.359297 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" Oct 30 20:26:03.359524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.359335 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" event={"ID":"915aea68-342e-4af2-9ea6-2f5d1ed3b84e","Type":"ContainerDied","Data":"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9"} Oct 30 20:26:03.359524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.359438 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn" event={"ID":"915aea68-342e-4af2-9ea6-2f5d1ed3b84e","Type":"ContainerDied","Data":"34da27c22244a4a424f6e3faa7b6db878694cd5d7086faf095ec54a6abcc2e82"} Oct 30 20:26:03.359524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.359462 5991 scope.go:122] "RemoveContainer" containerID="98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9" Oct 30 20:26:03.372086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.372057 5991 scope.go:122] "RemoveContainer" containerID="98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9" Oct 30 20:26:03.372517 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:03.372482 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9\": container with ID starting with 98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9 not found: ID does not exist" containerID="98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9" Oct 30 20:26:03.372595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.372532 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9"} err="failed to get container status \"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9\": rpc error: code = NotFound desc = could not find container \"98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9\": container with ID starting with 98701965b8201b1148fbd0f536f723fdab82c3cec26133f4d1347f494f39b6e9 not found: ID does not exist" Oct 30 20:26:03.414953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.414915 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:26:03.419485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:03.419451 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/auth-test-model-9d2720-kserve-77bd5f6cc7-72htn"] Oct 30 20:26:04.679172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:04.679130 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" path="/var/lib/kubelet/pods/915aea68-342e-4af2-9ea6-2f5d1ed3b84e/volumes" Oct 30 20:26:07.328572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:07.328522 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:26:12.390092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.390015 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" podUID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" containerName="payload-processing" containerID="cri-o://a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8" gracePeriod=25 Oct 30 20:26:12.720005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.719974 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:26:12.805335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.805296 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\" (UniqueName: \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\") pod \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " Oct 30 20:26:12.805522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.805482 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\") pod \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\" (UID: \"be800b0d-8e38-4d92-bbf6-e67b9af49f89\") " Oct 30 20:26:12.805878 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.805852 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume" pod "be800b0d-8e38-4d92-bbf6-e67b9af49f89" (UID: "be800b0d-8e38-4d92-bbf6-e67b9af49f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:12.807541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.807510 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m" pod "be800b0d-8e38-4d92-bbf6-e67b9af49f89" (UID: "be800b0d-8e38-4d92-bbf6-e67b9af49f89"). InnerVolumeSpecName "kube-api-access-l925m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:12.906684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.906643 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l925m\" (UniqueName: \"kubernetes.io/projected/be800b0d-8e38-4d92-bbf6-e67b9af49f89-kube-api-access-l925m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:12.906684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:12.906678 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be800b0d-8e38-4d92-bbf6-e67b9af49f89-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:13.433398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.433345 5991 generic.go:356] "Generic (PLEG): container finished" podID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" containerID="a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8" exitCode=0 Oct 30 20:26:13.433976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.433415 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" Oct 30 20:26:13.433976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.433434 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" event={"ID":"be800b0d-8e38-4d92-bbf6-e67b9af49f89","Type":"ContainerDied","Data":"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8"} Oct 30 20:26:13.433976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.433495 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578" event={"ID":"be800b0d-8e38-4d92-bbf6-e67b9af49f89","Type":"ContainerDied","Data":"2397d578fb737a55805306e24c153b9b04675308d204d3436fc2a23302bb3373"} Oct 30 20:26:13.433976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.433522 5991 scope.go:122] "RemoveContainer" containerID="a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8" Oct 30 20:26:13.444662 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.444639 5991 scope.go:122] "RemoveContainer" containerID="a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8" Oct 30 20:26:13.444930 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:13.444908 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8\": container with ID starting with a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8 not found: ID does not exist" containerID="a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8" Oct 30 20:26:13.445005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.444945 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8"} err="failed to get container status \"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8\": rpc error: code = NotFound desc = could not find container \"a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8\": container with ID starting with a7f0264c1a4dae129514f8c77c78e2a442f1b5205248d4fa80c52d5e7821bfa8 not found: ID does not exist" Oct 30 20:26:13.486825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.486782 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:26:13.498134 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:13.498102 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-76674ddf9d-5s578"] Oct 30 20:26:14.679528 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:14.679489 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" path="/var/lib/kubelet/pods/be800b0d-8e38-4d92-bbf6-e67b9af49f89/volumes" Oct 30 20:26:20.869300 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:20.869260 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:26:29.691354 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:29.691321 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:26:29.921681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:29.921650 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:26:33.993902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:33.993863 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 20:26:33.994413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:33.994139 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" containerID="cri-o://39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46" gracePeriod=30 Oct 30 20:26:39.012933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.012902 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:26:39.013264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.013127 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" containerID="cri-o://015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f" gracePeriod=30 Oct 30 20:26:39.278276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.278208 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 20:26:39.365565 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365529 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.365884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365613 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.365884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365692 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.365884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365868 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365987 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:39.366148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.365987 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:39.366148 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366073 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:39.366346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366328 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366441 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366400 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366444 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366484 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366617 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366601 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\") pod \"7336305a-07e9-4976-8ea0-65248a79a04d\" (UID: \"7336305a-07e9-4976-8ea0-65248a79a04d\") " Oct 30 20:26:39.366670 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.366612 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:39.367119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367095 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:39.367271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367192 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.367271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367209 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.367271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367223 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7336305a-07e9-4976-8ea0-65248a79a04d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.367271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367235 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.367271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.367247 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.368601 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.368571 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:39.369334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.369023 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:39.369518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.369492 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "kube-api-access-xnpbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:39.370156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.370130 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token" pod "7336305a-07e9-4976-8ea0-65248a79a04d" (UID: "7336305a-07e9-4976-8ea0-65248a79a04d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:39.468992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.468955 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpbp\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-kube-api-access-xnpbp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.468992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.468988 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7336305a-07e9-4976-8ea0-65248a79a04d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.468992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.468998 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7336305a-07e9-4976-8ea0-65248a79a04d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.469248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.469007 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7336305a-07e9-4976-8ea0-65248a79a04d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:39.630569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.630473 5991 generic.go:356] "Generic (PLEG): container finished" podID="7336305a-07e9-4976-8ea0-65248a79a04d" containerID="39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46" exitCode=0 Oct 30 20:26:39.630569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.630533 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" Oct 30 20:26:39.630569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.630534 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerDied","Data":"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46"} Oct 30 20:26:39.630817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.630618 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4" event={"ID":"7336305a-07e9-4976-8ea0-65248a79a04d","Type":"ContainerDied","Data":"c384fd3ce46dea809eacb9eb0f05a6108f072219e8cf9ec650c41f438d51c5f8"} Oct 30 20:26:39.630817 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.630641 5991 scope.go:122] "RemoveContainer" containerID="39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46" Oct 30 20:26:39.642589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.642570 5991 scope.go:122] "RemoveContainer" containerID="0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602" Oct 30 20:26:39.652876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.652855 5991 scope.go:122] "RemoveContainer" containerID="39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46" Oct 30 20:26:39.653201 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:39.653176 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46\": container with ID starting with 39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46 not found: ID does not exist" containerID="39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46" Oct 30 20:26:39.653336 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.653208 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46"} err="failed to get container status \"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46\": rpc error: code = NotFound desc = could not find container \"39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46\": container with ID starting with 39f8cce6603b70c7908bea0fbce1ecf420991d6391e2b3ff5fda1e3896c4cd46 not found: ID does not exist" Oct 30 20:26:39.653336 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.653226 5991 scope.go:122] "RemoveContainer" containerID="0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602" Oct 30 20:26:39.653519 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:39.653501 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602\": container with ID starting with 0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602 not found: ID does not exist" containerID="0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602" Oct 30 20:26:39.653572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.653525 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602"} err="failed to get container status \"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602\": rpc error: code = NotFound desc = could not find container \"0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602\": container with ID starting with 0ca3a004e57421f7a92335286919615065aff754f0c0e863c8083cc9c6718602 not found: ID does not exist" Oct 30 20:26:39.685808 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.685777 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 20:26:39.688498 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:39.688475 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-n2qg4"] Oct 30 20:26:40.680293 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:40.680259 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" path="/var/lib/kubelet/pods/7336305a-07e9-4976-8ea0-65248a79a04d/volumes" Oct 30 20:26:44.313701 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.313663 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:26:44.432294 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432260 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432350 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432476 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432519 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432549 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432600 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432608 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:44.432728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432665 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432946 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432734 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.432946 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.432808 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\") pod \"55e43405-b235-4f7d-9589-555f3d20bee4\" (UID: \"55e43405-b235-4f7d-9589-555f3d20bee4\") " Oct 30 20:26:44.433126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433102 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:44.433292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433260 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:44.433541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433512 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e43405-b235-4f7d-9589-555f3d20bee4-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.433541 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433541 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.433740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433555 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.433807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433744 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:44.433873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.433818 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:44.435840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.435808 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:44.435963 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.435839 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:44.435963 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.435915 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:44.436093 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.436042 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2" pod "55e43405-b235-4f7d-9589-555f3d20bee4" (UID: "55e43405-b235-4f7d-9589-555f3d20bee4"). InnerVolumeSpecName "kube-api-access-s75g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.534929 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e43405-b235-4f7d-9589-555f3d20bee4-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.534961 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.534970 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.534984 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75g2\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-kube-api-access-s75g2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.534999 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e43405-b235-4f7d-9589-555f3d20bee4-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.535014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.535012 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e43405-b235-4f7d-9589-555f3d20bee4-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:26:44.673496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.673458 5991 generic.go:356] "Generic (PLEG): container finished" podID="55e43405-b235-4f7d-9589-555f3d20bee4" containerID="015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f" exitCode=0 Oct 30 20:26:44.673680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.673569 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" Oct 30 20:26:44.680341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.680305 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" event={"ID":"55e43405-b235-4f7d-9589-555f3d20bee4","Type":"ContainerDied","Data":"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f"} Oct 30 20:26:44.680507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.680356 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj" event={"ID":"55e43405-b235-4f7d-9589-555f3d20bee4","Type":"ContainerDied","Data":"5f0dfdbd710b41c1443a15a6767725f59fe4c8b843f0fdc92d8630623ff10163"} Oct 30 20:26:44.680507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.680425 5991 scope.go:122] "RemoveContainer" containerID="015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f" Oct 30 20:26:44.693265 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.693244 5991 scope.go:122] "RemoveContainer" containerID="015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f" Oct 30 20:26:44.693575 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:44.693552 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f\": container with ID starting with 015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f not found: ID does not exist" containerID="015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f" Oct 30 20:26:44.693676 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.693585 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f"} err="failed to get container status \"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f\": rpc error: code = NotFound desc = could not find container \"015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f\": container with ID starting with 015f8c0d7345ffc4bc0340ca84f447b348523263eaa4292c0c31929ac0b1792f not found: ID does not exist" Oct 30 20:26:44.731385 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.731337 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:26:44.744010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.743979 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-r67bj"] Oct 30 20:26:44.799998 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.799909 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:26:44.800664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800648 5991 cpu_manager.go:414] "containerMap: removing container" podUID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" containerName="payload-pre-processing" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800666 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" containerName="payload-pre-processing" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800680 5991 cpu_manager.go:414] "containerMap: removing container" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800686 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800697 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800702 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800709 5991 cpu_manager.go:414] "containerMap: removing container" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" Oct 30 20:26:44.800713 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800713 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800724 5991 cpu_manager.go:414] "containerMap: removing container" podUID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" containerName="main" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800728 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" containerName="main" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800741 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800746 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800757 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800761 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800772 5991 cpu_manager.go:414] "containerMap: removing container" podUID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" containerName="payload-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800776 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" containerName="payload-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800784 5991 cpu_manager.go:414] "containerMap: removing container" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800788 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800795 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800799 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800813 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800817 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800825 5991 cpu_manager.go:414] "containerMap: removing container" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800830 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800841 5991 cpu_manager.go:414] "containerMap: removing container" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800845 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800852 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7bae6318-f6d7-4874-a331-784d18707f48" containerName="payload-pre-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800856 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bae6318-f6d7-4874-a331-784d18707f48" containerName="payload-pre-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800863 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" containerName="payload-pre-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800867 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" containerName="payload-pre-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800878 5991 cpu_manager.go:414] "containerMap: removing container" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800883 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800890 5991 cpu_manager.go:414] "containerMap: removing container" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" Oct 30 20:26:44.800937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800894 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800988 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f6375d-26ca-4195-91aa-56bfd1a3a4ab" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.800999 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801006 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bac318-6c9d-4df0-97fa-8a123eddda8c" containerName="main" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801014 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bae6318-f6d7-4874-a331-784d18707f48" containerName="payload-pre-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801021 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5aef768-0b9a-43fa-aeb8-a45744dc2830" containerName="payload-pre-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801028 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="605ab214-f577-4cd3-aa4b-2794c2c8f21d" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801033 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="66499969-3c1f-4a18-816c-a30d6e470abb" containerName="payload-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801041 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dff2bea-e37c-450e-98ad-109c2b70f6f3" containerName="payload-pre-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801046 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8af32d-b630-45ce-93e9-a1df20cff8b3" containerName="main" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801051 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42b2133-af7b-4713-966b-e98f5e14e3f4" containerName="main" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801055 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e12231-4859-49d4-9c38-d485230c0a6b" containerName="main" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801061 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="915aea68-342e-4af2-9ea6-2f5d1ed3b84e" containerName="main" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801067 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801073 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801079 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="be800b0d-8e38-4d92-bbf6-e67b9af49f89" containerName="payload-processing" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801085 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfb0705-0c49-4e44-a2af-f3c042ad6662" containerName="istio-proxy" Oct 30 20:26:44.801785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.801091 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7336305a-07e9-4976-8ea0-65248a79a04d" containerName="istio-proxy" Oct 30 20:26:44.806167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.806142 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:44.825587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.825555 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c19563\"" Oct 30 20:26:44.825587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.825557 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c19563-dockercfg-z5x49\"" Oct 30 20:26:44.830004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.829978 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:26:44.940659 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.940619 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjrp\" (UniqueName: \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:44.940856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:44.940683 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.041750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.041718 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjrp\" (UniqueName: \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.041966 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.041802 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.042698 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.042665 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.062404 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.062299 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjrp\" (UniqueName: \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\") pod \"payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.120294 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.120257 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.437515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.437466 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:26:45.438781 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:26:45.438747 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf59d3d3_c326_422e_8033_600514a97a46.slice/crio-c8eb7345ea553828eb76c3017384ffe88a28f4061d21a5f9817be65e7aa5158e WatchSource:0}: Error finding container c8eb7345ea553828eb76c3017384ffe88a28f4061d21a5f9817be65e7aa5158e: Status 404 returned error can't find the container with id c8eb7345ea553828eb76c3017384ffe88a28f4061d21a5f9817be65e7aa5158e Oct 30 20:26:45.683268 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.683229 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" event={"ID":"df59d3d3-c326-422e-8033-600514a97a46","Type":"ContainerStarted","Data":"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de"} Oct 30 20:26:45.683522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.683290 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" event={"ID":"df59d3d3-c326-422e-8033-600514a97a46","Type":"ContainerStarted","Data":"c8eb7345ea553828eb76c3017384ffe88a28f4061d21a5f9817be65e7aa5158e"} Oct 30 20:26:45.683522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.683425 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:26:45.726047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:45.725991 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" podStartSLOduration=1.7259776260000002 podStartE2EDuration="1.725977626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:45.724721179 +0000 UTC m=+3073.174213904" watchObservedRunningTime="2026-10-30 20:26:45.725977626 +0000 UTC m=+3073.175470351" Oct 30 20:26:46.680921 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:46.680884 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e43405-b235-4f7d-9589-555f3d20bee4" path="/var/lib/kubelet/pods/55e43405-b235-4f7d-9589-555f3d20bee4/volumes" Oct 30 20:26:50.871728 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:26:50.871685 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:26:54.054195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:54.054157 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:26:56.379830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:56.379795 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 20:26:56.380258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:56.379995 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" containerID="cri-o://13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3" gracePeriod=30 Oct 30 20:26:56.698126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:26:56.698043 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:27:01.655952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.655870 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 20:27:01.787250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787218 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787282 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787339 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787469 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787512 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787582 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787613 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787741 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787746 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:01.787804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787652 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:01.787973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787812 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\") pod \"e27c4352-5fd8-45e6-ad8f-2771530b9100\" (UID: \"e27c4352-5fd8-45e6-ad8f-2771530b9100\") " Oct 30 20:27:01.787973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787907 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:01.788082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.787958 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:01.788082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788015 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:01.788575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788551 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.788760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788732 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.788884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788763 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.788884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788778 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.788884 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.788790 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27c4352-5fd8-45e6-ad8f-2771530b9100-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.790066 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.790040 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:01.790162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.790082 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:27:01.790162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.790099 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:01.790484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.790465 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9" pod "e27c4352-5fd8-45e6-ad8f-2771530b9100" (UID: "e27c4352-5fd8-45e6-ad8f-2771530b9100"). InnerVolumeSpecName "kube-api-access-szzq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:01.804810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.804784 5991 generic.go:356] "Generic (PLEG): container finished" podID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerID="13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3" exitCode=0 Oct 30 20:27:01.804903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.804827 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" Oct 30 20:27:01.804903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.804851 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" event={"ID":"e27c4352-5fd8-45e6-ad8f-2771530b9100","Type":"ContainerDied","Data":"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3"} Oct 30 20:27:01.804903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.804895 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d" event={"ID":"e27c4352-5fd8-45e6-ad8f-2771530b9100","Type":"ContainerDied","Data":"6a6e41afb84a55799edfd066df8af84b65e95a1c0968c0115953518c40c5244f"} Oct 30 20:27:01.805013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.804926 5991 scope.go:122] "RemoveContainer" containerID="13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3" Oct 30 20:27:01.822208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.822191 5991 scope.go:122] "RemoveContainer" containerID="13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3" Oct 30 20:27:01.822481 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:27:01.822462 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3\": container with ID starting with 13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3 not found: ID does not exist" containerID="13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3" Oct 30 20:27:01.822579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.822487 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3"} err="failed to get container status \"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3\": rpc error: code = NotFound desc = could not find container \"13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3\": container with ID starting with 13af3e99c0a12d273f3193643c52461ff2343e926a9164c05cc7d0964ca3a4d3 not found: ID does not exist" Oct 30 20:27:01.859683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.859646 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 20:27:01.864402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.864354 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6j42d"] Oct 30 20:27:01.889961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.889930 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.889961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.889955 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.889961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.889965 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzq9\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-kube-api-access-szzq9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:01.890169 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:01.889975 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27c4352-5fd8-45e6-ad8f-2771530b9100-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:02.681700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:02.681666 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" path="/var/lib/kubelet/pods/e27c4352-5fd8-45e6-ad8f-2771530b9100/volumes" Oct 30 20:27:10.932076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:10.932040 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:27:10.932906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:10.932869 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" podUID="c265a09c-4ada-46f6-ab57-8fe19794f506" containerName="maas-api" containerID="cri-o://40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd" gracePeriod=30 Oct 30 20:27:11.126758 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.126718 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:27:11.127342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.127311 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" podUID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" containerName="payload-pre-processing" containerID="cri-o://9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6" gracePeriod=30 Oct 30 20:27:11.269041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.269004 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:27:11.416243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.416216 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:27:11.427454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.427418 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\") pod \"c265a09c-4ada-46f6-ab57-8fe19794f506\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " Oct 30 20:27:11.427648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.427478 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\") pod \"c265a09c-4ada-46f6-ab57-8fe19794f506\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " Oct 30 20:27:11.427726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.427671 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\" (UniqueName: \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\") pod \"c265a09c-4ada-46f6-ab57-8fe19794f506\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " Oct 30 20:27:11.427726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.427697 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\") pod \"c265a09c-4ada-46f6-ab57-8fe19794f506\" (UID: \"c265a09c-4ada-46f6-ab57-8fe19794f506\") " Oct 30 20:27:11.428165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.428132 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca" pod "c265a09c-4ada-46f6-ab57-8fe19794f506" (UID: "c265a09c-4ada-46f6-ab57-8fe19794f506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:11.429731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.429705 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95" pod "c265a09c-4ada-46f6-ab57-8fe19794f506" (UID: "c265a09c-4ada-46f6-ab57-8fe19794f506"). InnerVolumeSpecName "kube-api-access-sqk95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:11.429824 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.429717 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls" pod "c265a09c-4ada-46f6-ab57-8fe19794f506" (UID: "c265a09c-4ada-46f6-ab57-8fe19794f506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:11.429824 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.429732 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs" pod "c265a09c-4ada-46f6-ab57-8fe19794f506" (UID: "c265a09c-4ada-46f6-ab57-8fe19794f506"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:11.529514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529428 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\") pod \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " Oct 30 20:27:11.529514 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529469 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\" (UniqueName: \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\") pod \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\" (UID: \"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f\") " Oct 30 20:27:11.529813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529784 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqk95\" (UniqueName: \"kubernetes.io/projected/c265a09c-4ada-46f6-ab57-8fe19794f506-kube-api-access-sqk95\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.529813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529796 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume" pod "5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" (UID: "5ed9b983-3fc1-487a-9cd0-2a4660f5db7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:11.529975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529804 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c265a09c-4ada-46f6-ab57-8fe19794f506-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.529975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529848 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.529975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.529859 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c265a09c-4ada-46f6-ab57-8fe19794f506-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.531548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.531524 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk" pod "5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" (UID: "5ed9b983-3fc1-487a-9cd0-2a4660f5db7f"). InnerVolumeSpecName "kube-api-access-crbmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:11.631314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.631275 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.631314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.631310 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbmk\" (UniqueName: \"kubernetes.io/projected/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f-kube-api-access-crbmk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:11.876591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.876505 5991 generic.go:356] "Generic (PLEG): container finished" podID="c265a09c-4ada-46f6-ab57-8fe19794f506" containerID="40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd" exitCode=0 Oct 30 20:27:11.876591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.876561 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" Oct 30 20:27:11.876840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.876587 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" event={"ID":"c265a09c-4ada-46f6-ab57-8fe19794f506","Type":"ContainerDied","Data":"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd"} Oct 30 20:27:11.876840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.876637 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764" event={"ID":"c265a09c-4ada-46f6-ab57-8fe19794f506","Type":"ContainerDied","Data":"88a2803da2d3a22e447e177479fcfb548535faade75bea924013ea0cae1ebcc8"} Oct 30 20:27:11.876840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.876659 5991 scope.go:122] "RemoveContainer" containerID="40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd" Oct 30 20:27:11.878217 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.878192 5991 generic.go:356] "Generic (PLEG): container finished" podID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" containerID="9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6" exitCode=0 Oct 30 20:27:11.878338 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.878238 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" Oct 30 20:27:11.878338 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.878276 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" event={"ID":"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f","Type":"ContainerDied","Data":"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6"} Oct 30 20:27:11.878338 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.878315 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9" event={"ID":"5ed9b983-3fc1-487a-9cd0-2a4660f5db7f","Type":"ContainerDied","Data":"eb5aa57049badba7501434eda77184c528c3b82de872583b00d916e022857dec"} Oct 30 20:27:11.888733 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.888713 5991 scope.go:122] "RemoveContainer" containerID="40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd" Oct 30 20:27:11.889034 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:27:11.889013 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd\": container with ID starting with 40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd not found: ID does not exist" containerID="40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd" Oct 30 20:27:11.889101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.889043 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd"} err="failed to get container status \"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd\": rpc error: code = NotFound desc = could not find container \"40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd\": container with ID starting with 40f5db3be38e5057b831aa6598cc2969e36ee08a83d51c3bfcb96954e38059fd not found: ID does not exist" Oct 30 20:27:11.889101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.889060 5991 scope.go:122] "RemoveContainer" containerID="9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6" Oct 30 20:27:11.899277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.899259 5991 scope.go:122] "RemoveContainer" containerID="9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6" Oct 30 20:27:11.899598 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:27:11.899576 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6\": container with ID starting with 9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6 not found: ID does not exist" containerID="9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6" Oct 30 20:27:11.899668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.899604 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6"} err="failed to get container status \"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6\": rpc error: code = NotFound desc = could not find container \"9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6\": container with ID starting with 9eb940a3ce4a480f5e297365b1cdfce25311c287371236afa7bd2a3253026fa6 not found: ID does not exist" Oct 30 20:27:11.928788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.928757 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:27:11.939809 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.939780 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-188d45-6f95bbd798-zl764"] Oct 30 20:27:11.973420 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.973386 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:27:11.979168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:11.979135 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-7dfb8d544c-gr8s9"] Oct 30 20:27:12.678822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:12.678784 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" path="/var/lib/kubelet/pods/5ed9b983-3fc1-487a-9cd0-2a4660f5db7f/volumes" Oct 30 20:27:12.679277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:12.679262 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c265a09c-4ada-46f6-ab57-8fe19794f506" path="/var/lib/kubelet/pods/c265a09c-4ada-46f6-ab57-8fe19794f506/volumes" Oct 30 20:27:18.659750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.659704 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:27:18.660549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.659924 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" podUID="5105e7d1-170e-4548-a210-3b53c12e2abf" containerName="payload-processing" containerID="cri-o://b15cbd4589155dfa9fade002cc6cd5d86fd78264265fbd30996dc1c804648700" gracePeriod=30 Oct 30 20:27:18.701141 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.701107 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:27:18.701358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.701319 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" podUID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" containerName="payload-pre-processing" containerID="cri-o://2b46f5f0501a13b33f0e08a4fbb4f1b92099f8d6fb5736de91327aef66bb1099" gracePeriod=30 Oct 30 20:27:18.932926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.932831 5991 generic.go:356] "Generic (PLEG): container finished" podID="5105e7d1-170e-4548-a210-3b53c12e2abf" containerID="b15cbd4589155dfa9fade002cc6cd5d86fd78264265fbd30996dc1c804648700" exitCode=0 Oct 30 20:27:18.933106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.932930 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" event={"ID":"5105e7d1-170e-4548-a210-3b53c12e2abf","Type":"ContainerDied","Data":"b15cbd4589155dfa9fade002cc6cd5d86fd78264265fbd30996dc1c804648700"} Oct 30 20:27:18.935256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.935220 5991 generic.go:356] "Generic (PLEG): container finished" podID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" containerID="2b46f5f0501a13b33f0e08a4fbb4f1b92099f8d6fb5736de91327aef66bb1099" exitCode=0 Oct 30 20:27:18.935459 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.935343 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" event={"ID":"244ecbe2-ab8c-4c3f-ae48-57803ece337c","Type":"ContainerDied","Data":"2b46f5f0501a13b33f0e08a4fbb4f1b92099f8d6fb5736de91327aef66bb1099"} Oct 30 20:27:18.998325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:18.998295 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:27:19.036523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.036494 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:27:19.123953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.123909 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\") pod \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " Oct 30 20:27:19.124151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124113 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\" (UniqueName: \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\") pod \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\" (UID: \"244ecbe2-ab8c-4c3f-ae48-57803ece337c\") " Oct 30 20:27:19.124198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124160 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\" (UniqueName: \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\") pod \"5105e7d1-170e-4548-a210-3b53c12e2abf\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " Oct 30 20:27:19.124274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124258 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\") pod \"5105e7d1-170e-4548-a210-3b53c12e2abf\" (UID: \"5105e7d1-170e-4548-a210-3b53c12e2abf\") " Oct 30 20:27:19.124401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124337 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume" pod "244ecbe2-ab8c-4c3f-ae48-57803ece337c" (UID: "244ecbe2-ab8c-4c3f-ae48-57803ece337c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:19.124694 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124642 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume" pod "5105e7d1-170e-4548-a210-3b53c12e2abf" (UID: "5105e7d1-170e-4548-a210-3b53c12e2abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:19.124885 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124864 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5105e7d1-170e-4548-a210-3b53c12e2abf-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:19.124970 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.124891 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244ecbe2-ab8c-4c3f-ae48-57803ece337c-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:19.126563 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.126540 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv" pod "244ecbe2-ab8c-4c3f-ae48-57803ece337c" (UID: "244ecbe2-ab8c-4c3f-ae48-57803ece337c"). InnerVolumeSpecName "kube-api-access-5ptfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:19.126675 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.126571 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn" pod "5105e7d1-170e-4548-a210-3b53c12e2abf" (UID: "5105e7d1-170e-4548-a210-3b53c12e2abf"). InnerVolumeSpecName "kube-api-access-nxxhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:19.225825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.225797 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptfv\" (UniqueName: \"kubernetes.io/projected/244ecbe2-ab8c-4c3f-ae48-57803ece337c-kube-api-access-5ptfv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:19.225825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.225823 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxhn\" (UniqueName: \"kubernetes.io/projected/5105e7d1-170e-4548-a210-3b53c12e2abf-kube-api-access-nxxhn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:27:19.943610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.943525 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" Oct 30 20:27:19.943610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.943567 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5" event={"ID":"5105e7d1-170e-4548-a210-3b53c12e2abf","Type":"ContainerDied","Data":"73c79017639085bee5fbe800e176d8fbdbb672928cead60b3404bff3507fe50d"} Oct 30 20:27:19.944150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.943643 5991 scope.go:122] "RemoveContainer" containerID="b15cbd4589155dfa9fade002cc6cd5d86fd78264265fbd30996dc1c804648700" Oct 30 20:27:19.945064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.945046 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" Oct 30 20:27:19.945163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.945061 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4" event={"ID":"244ecbe2-ab8c-4c3f-ae48-57803ece337c","Type":"ContainerDied","Data":"c3893005e54cc86a1f104ec854358a6a73ce2ee3112bc96142916ef756e7a5c1"} Oct 30 20:27:19.955702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.955679 5991 scope.go:122] "RemoveContainer" containerID="2b46f5f0501a13b33f0e08a4fbb4f1b92099f8d6fb5736de91327aef66bb1099" Oct 30 20:27:19.994922 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:19.994890 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:27:20.006917 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:20.006888 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-7d8f6b9848-sntp5"] Oct 30 20:27:20.030910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:20.030879 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:27:20.034951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:20.034923 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-98759647f-2qkv4"] Oct 30 20:27:20.680200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:20.680162 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" path="/var/lib/kubelet/pods/244ecbe2-ab8c-4c3f-ae48-57803ece337c/volumes" Oct 30 20:27:20.680658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:20.680641 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5105e7d1-170e-4548-a210-3b53c12e2abf" path="/var/lib/kubelet/pods/5105e7d1-170e-4548-a210-3b53c12e2abf/volumes" Oct 30 20:27:20.870784 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:27:20.870744 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:27:28.057429 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:28.057384 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:27:41.059973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.059942 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:27:41.060702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060685 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c265a09c-4ada-46f6-ab57-8fe19794f506" containerName="maas-api" Oct 30 20:27:41.060702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060702 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c265a09c-4ada-46f6-ab57-8fe19794f506" containerName="maas-api" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060724 5991 cpu_manager.go:414] "containerMap: removing container" podUID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" containerName="payload-pre-processing" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060730 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" containerName="payload-pre-processing" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060745 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" containerName="payload-pre-processing" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060750 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" containerName="payload-pre-processing" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060776 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060781 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060787 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5105e7d1-170e-4548-a210-3b53c12e2abf" containerName="payload-processing" Oct 30 20:27:41.060804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060792 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5105e7d1-170e-4548-a210-3b53c12e2abf" containerName="payload-processing" Oct 30 20:27:41.061052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060907 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c265a09c-4ada-46f6-ab57-8fe19794f506" containerName="maas-api" Oct 30 20:27:41.061052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060917 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="244ecbe2-ab8c-4c3f-ae48-57803ece337c" containerName="payload-pre-processing" Oct 30 20:27:41.061052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060928 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed9b983-3fc1-487a-9cd0-2a4660f5db7f" containerName="payload-pre-processing" Oct 30 20:27:41.061052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060939 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27c4352-5fd8-45e6-ad8f-2771530b9100" containerName="istio-proxy" Oct 30 20:27:41.061052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.060945 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5105e7d1-170e-4548-a210-3b53c12e2abf" containerName="payload-processing" Oct 30 20:27:41.066439 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.066413 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.084931 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.084894 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:27:41.093788 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.093751 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-843ea8-serving-cert\"" Oct 30 20:27:41.118478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.118426 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphtq\" (UniqueName: \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.118715 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.118520 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.118802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.118722 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.118802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.118764 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.220106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.220068 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.220308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.220118 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.220308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.220168 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphtq\" (UniqueName: \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.220308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.220213 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.221136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.221109 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.222751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.222728 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.222877 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.222859 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.240440 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.240408 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphtq\" (UniqueName: \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\") pod \"maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.381004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.380900 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:41.689017 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.688979 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:27:41.689163 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:27:41.689029 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bc2b6c_ce87_44fb_9d69_25edfbe57f0f.slice/crio-4e3b5c731efaefa10346eba12eff8dce5a1e226781c70779acf4256bf2a42d37 WatchSource:0}: Error finding container 4e3b5c731efaefa10346eba12eff8dce5a1e226781c70779acf4256bf2a42d37: Status 404 returned error can't find the container with id 4e3b5c731efaefa10346eba12eff8dce5a1e226781c70779acf4256bf2a42d37 Oct 30 20:27:41.690511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.690489 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:27:41.770358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.770316 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:27:41.776755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.776724 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.793144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.793115 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5c875d-openshift-default-dockercfg-crmft\"" Oct 30 20:27:41.808305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.808258 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:27:41.835247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835206 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835504 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835344 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835504 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835430 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp2p\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835654 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835540 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835654 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835635 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835770 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835657 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835770 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835675 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835770 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835728 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.835934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.835793 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.936902 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.936952 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.936978 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937029 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937059 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937166 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937285 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937357 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp2p\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937420 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937572 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937478 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937657 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937859 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937833 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.937930 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.937837 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.938311 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.938287 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.939852 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.939826 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.939987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.939968 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.957966 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.957935 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp2p\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:41.962024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:41.961991 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:42.093710 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.093677 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:42.099653 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.099607 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" event={"ID":"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f","Type":"ContainerStarted","Data":"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641"} Oct 30 20:27:42.099772 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.099677 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" event={"ID":"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f","Type":"ContainerStarted","Data":"4e3b5c731efaefa10346eba12eff8dce5a1e226781c70779acf4256bf2a42d37"} Oct 30 20:27:42.099835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.099794 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:42.144136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.144077 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" podStartSLOduration=0.88087569 podStartE2EDuration="1.144063558s" totalImagesPullingTime="263.187868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:42.142099426 +0000 UTC m=+3129.591592158" watchObservedRunningTime="2026-10-30 20:27:42.144063558 +0000 UTC m=+3129.593556283" Oct 30 20:27:42.414345 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:27:42.414311 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c07ee87_58d0_4abe_8f10_8781cdba3f77.slice/crio-2954c2653ffc46ebc107c6c227f7e63eda22848b3b0b5792fca1180b76af9b5c WatchSource:0}: Error finding container 2954c2653ffc46ebc107c6c227f7e63eda22848b3b0b5792fca1180b76af9b5c: Status 404 returned error can't find the container with id 2954c2653ffc46ebc107c6c227f7e63eda22848b3b0b5792fca1180b76af9b5c Oct 30 20:27:42.414794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.414767 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:27:42.417674 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.417636 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:27:42.417785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.417707 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:27:42.417785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:42.417736 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:27:43.107570 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:43.107530 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" event={"ID":"5c07ee87-58d0-4abe-8f10-8781cdba3f77","Type":"ContainerStarted","Data":"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88"} Oct 30 20:27:43.108095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:43.107585 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" event={"ID":"5c07ee87-58d0-4abe-8f10-8781cdba3f77","Type":"ContainerStarted","Data":"2954c2653ffc46ebc107c6c227f7e63eda22848b3b0b5792fca1180b76af9b5c"} Oct 30 20:27:43.165859 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:43.165804 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podStartSLOduration=2.16579058 podStartE2EDuration="2.16579058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:43.164338747 +0000 UTC m=+3130.613831695" watchObservedRunningTime="2026-10-30 20:27:43.16579058 +0000 UTC m=+3130.615283305" Oct 30 20:27:44.094864 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:44.094826 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:44.096500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:44.096472 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:44.096651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:44.096523 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:45.094076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:45.094033 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:45.094609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:45.094099 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:46.094539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:46.094497 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:46.095001 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:46.094555 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:47.095119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:47.095072 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:47.095600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:47.095137 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:48.094946 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:48.094901 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:48.095154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:48.094964 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:48.114011 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:48.113982 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:27:49.095082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:49.095032 5991 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:27:49.095307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:49.095126 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 20:27:50.100275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:50.100242 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:50.100735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:50.100714 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:50.101464 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:27:50.101444 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:27:50.870939 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:27:50.870899 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:28:08.113185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:08.113149 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:28:20.871169 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:28:20.871079 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:28:37.966314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:37.966277 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:28:37.966830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:37.966504 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" podUID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" containerName="maas-api" containerID="cri-o://e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28" gracePeriod=30 Oct 30 20:28:38.133271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.133227 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:28:38.133519 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.133495 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" podUID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" containerName="payload-processing" containerID="cri-o://1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302" gracePeriod=30 Oct 30 20:28:38.185680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.185237 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:28:38.186483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.186049 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" podUID="8e186910-179a-4b33-bc70-1787f40b66c3" containerName="payload-pre-processing" containerID="cri-o://bef0df7fcbf3d1df8f0e40eaaffb4f1134ed1d81f9c756d7162f884be900a49e" gracePeriod=30 Oct 30 20:28:38.333236 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.333201 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:28:38.426801 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.426775 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\") pod \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " Oct 30 20:28:38.426920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.426844 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\") pod \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " Oct 30 20:28:38.426980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.426946 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\") pod \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " Oct 30 20:28:38.427167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.427151 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\" (UniqueName: \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\") pod \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\" (UID: \"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae\") " Oct 30 20:28:38.427434 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.427399 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca" pod "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" (UID: "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:38.427719 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.427698 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.430061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.430028 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk" pod "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" (UID: "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae"). InnerVolumeSpecName "kube-api-access-xgbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:38.430178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.430110 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls" pod "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" (UID: "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:38.430274 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.430248 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs" pod "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" (UID: "d13bb142-e034-4f7c-8a54-fdc3c7cab4ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:38.491111 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.491030 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:28:38.508641 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.508604 5991 generic.go:356] "Generic (PLEG): container finished" podID="8e186910-179a-4b33-bc70-1787f40b66c3" containerID="bef0df7fcbf3d1df8f0e40eaaffb4f1134ed1d81f9c756d7162f884be900a49e" exitCode=0 Oct 30 20:28:38.508910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.508647 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" event={"ID":"8e186910-179a-4b33-bc70-1787f40b66c3","Type":"ContainerDied","Data":"bef0df7fcbf3d1df8f0e40eaaffb4f1134ed1d81f9c756d7162f884be900a49e"} Oct 30 20:28:38.511598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.511261 5991 generic.go:356] "Generic (PLEG): container finished" podID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" containerID="e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28" exitCode=0 Oct 30 20:28:38.511598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.511474 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" event={"ID":"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae","Type":"ContainerDied","Data":"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28"} Oct 30 20:28:38.511598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.511530 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" event={"ID":"d13bb142-e034-4f7c-8a54-fdc3c7cab4ae","Type":"ContainerDied","Data":"1f79ea5377d3aeb3dcfef07110dc37faf5f36ece9315f75e521c04732f39f79d"} Oct 30 20:28:38.511598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.511562 5991 scope.go:122] "RemoveContainer" containerID="e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28" Oct 30 20:28:38.511938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.511749 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t" Oct 30 20:28:38.517477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.517430 5991 generic.go:356] "Generic (PLEG): container finished" podID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" containerID="1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302" exitCode=0 Oct 30 20:28:38.517625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.517521 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" event={"ID":"5f0244bf-ad49-4a98-b3b0-e95f9c280961","Type":"ContainerDied","Data":"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302"} Oct 30 20:28:38.517625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.517573 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" event={"ID":"5f0244bf-ad49-4a98-b3b0-e95f9c280961","Type":"ContainerDied","Data":"5995b9dcc775303757200c511ae9ad5819b738b2c6bfea85b5d0b7dd5ab24099"} Oct 30 20:28:38.517752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.517648 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv" Oct 30 20:28:38.532032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.529337 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\" (UniqueName: \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\") pod \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " Oct 30 20:28:38.533446 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.533197 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume" pod "5f0244bf-ad49-4a98-b3b0-e95f9c280961" (UID: "5f0244bf-ad49-4a98-b3b0-e95f9c280961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:38.535808 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.533764 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\") pod \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\" (UID: \"5f0244bf-ad49-4a98-b3b0-e95f9c280961\") " Oct 30 20:28:38.537022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.536982 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbhk\" (UniqueName: \"kubernetes.io/projected/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-kube-api-access-xgbhk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.537022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.537022 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.537259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.537052 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.537259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.537065 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0244bf-ad49-4a98-b3b0-e95f9c280961-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.542108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.542065 5991 scope.go:122] "RemoveContainer" containerID="e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28" Oct 30 20:28:38.546006 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:28:38.543438 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28\": container with ID starting with e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28 not found: ID does not exist" containerID="e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28" Oct 30 20:28:38.546006 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.543508 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28"} err="failed to get container status \"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28\": rpc error: code = NotFound desc = could not find container \"e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28\": container with ID starting with e93ac64467293e1e2e3a5ca62e0f61a4b0690a6cbdda73584bdadf0782e74a28 not found: ID does not exist" Oct 30 20:28:38.546006 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.543542 5991 scope.go:122] "RemoveContainer" containerID="1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302" Oct 30 20:28:38.550492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.549844 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:28:38.550492 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.549902 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j" pod "5f0244bf-ad49-4a98-b3b0-e95f9c280961" (UID: "5f0244bf-ad49-4a98-b3b0-e95f9c280961"). InnerVolumeSpecName "kube-api-access-msq7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:38.563722 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.563692 5991 scope.go:122] "RemoveContainer" containerID="1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302" Oct 30 20:28:38.564140 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:28:38.564113 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302\": container with ID starting with 1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302 not found: ID does not exist" containerID="1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302" Oct 30 20:28:38.564241 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.564152 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302"} err="failed to get container status \"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302\": rpc error: code = NotFound desc = could not find container \"1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302\": container with ID starting with 1400cac0f5ec354c6b2fedf193677e0a67f8952fc80422784284fd53dc1dc302 not found: ID does not exist" Oct 30 20:28:38.637615 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.637567 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\" (UniqueName: \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\") pod \"8e186910-179a-4b33-bc70-1787f40b66c3\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " Oct 30 20:28:38.637833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.637640 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\") pod \"8e186910-179a-4b33-bc70-1787f40b66c3\" (UID: \"8e186910-179a-4b33-bc70-1787f40b66c3\") " Oct 30 20:28:38.638502 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.638475 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msq7j\" (UniqueName: \"kubernetes.io/projected/5f0244bf-ad49-4a98-b3b0-e95f9c280961-kube-api-access-msq7j\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.638920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.638282 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume" pod "8e186910-179a-4b33-bc70-1787f40b66c3" (UID: "8e186910-179a-4b33-bc70-1787f40b66c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:38.640101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.640071 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq" pod "8e186910-179a-4b33-bc70-1787f40b66c3" (UID: "8e186910-179a-4b33-bc70-1787f40b66c3"). InnerVolumeSpecName "kube-api-access-4g2vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:38.643581 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.643557 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:28:38.649569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.649544 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30e92c-57768596f-7rf8t"] Oct 30 20:28:38.680577 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.680532 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" path="/var/lib/kubelet/pods/d13bb142-e034-4f7c-8a54-fdc3c7cab4ae/volumes" Oct 30 20:28:38.740200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.740167 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2vq\" (UniqueName: \"kubernetes.io/projected/8e186910-179a-4b33-bc70-1787f40b66c3-kube-api-access-4g2vq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.740200 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.740195 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e186910-179a-4b33-bc70-1787f40b66c3-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:38.858399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.858261 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:28:38.860923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:38.860892 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-68777b649b-sgmzv"] Oct 30 20:28:39.526778 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:39.526743 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" Oct 30 20:28:39.527271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:39.526780 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv" event={"ID":"8e186910-179a-4b33-bc70-1787f40b66c3","Type":"ContainerDied","Data":"6045eab4a4bd996519fc7c738e2b98521a0776ea046f4a03ae96154226a78b4c"} Oct 30 20:28:39.527271 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:39.526837 5991 scope.go:122] "RemoveContainer" containerID="bef0df7fcbf3d1df8f0e40eaaffb4f1134ed1d81f9c756d7162f884be900a49e" Oct 30 20:28:39.582948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:39.582911 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:28:39.595448 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:39.595402 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30e92c-6d76586df9-j66sv"] Oct 30 20:28:40.683326 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:40.683291 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" path="/var/lib/kubelet/pods/5f0244bf-ad49-4a98-b3b0-e95f9c280961/volumes" Oct 30 20:28:40.683799 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:40.683769 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e186910-179a-4b33-bc70-1787f40b66c3" path="/var/lib/kubelet/pods/8e186910-179a-4b33-bc70-1787f40b66c3/volumes" Oct 30 20:28:50.879601 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:28:50.879558 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:28:51.089041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089002 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:28:51.089784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089762 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" containerName="payload-processing" Oct 30 20:28:51.089784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089781 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" containerName="payload-processing" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089816 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8e186910-179a-4b33-bc70-1787f40b66c3" containerName="payload-pre-processing" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089821 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e186910-179a-4b33-bc70-1787f40b66c3" containerName="payload-pre-processing" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089832 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" containerName="maas-api" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089837 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" containerName="maas-api" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089944 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e186910-179a-4b33-bc70-1787f40b66c3" containerName="payload-pre-processing" Oct 30 20:28:51.089961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089955 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13bb142-e034-4f7c-8a54-fdc3c7cab4ae" containerName="maas-api" Oct 30 20:28:51.090178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.089968 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0244bf-ad49-4a98-b3b0-e95f9c280961" containerName="payload-processing" Oct 30 20:28:51.095189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.095168 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.110285 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.110250 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.110686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.110662 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jg4\" (UniqueName: \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.122168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.122135 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-843ea8\"" Oct 30 20:28:51.122402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.122185 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-843ea8-dockercfg-z7qcv\"" Oct 30 20:28:51.126495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.126471 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:28:51.211823 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.211737 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jg4\" (UniqueName: \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.211987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.211894 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.212635 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.212613 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.231021 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.230993 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jg4\" (UniqueName: \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\") pod \"payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.409407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.409374 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:51.716964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:51.716851 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:28:51.718520 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:28:51.718445 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ef9ec4_a498_4de2_a9ac_368e26e5f67e.slice/crio-053e212820b2c5df5e2fe65182784af985099e12afa56905d0667c16148806c3 WatchSource:0}: Error finding container 053e212820b2c5df5e2fe65182784af985099e12afa56905d0667c16148806c3: Status 404 returned error can't find the container with id 053e212820b2c5df5e2fe65182784af985099e12afa56905d0667c16148806c3 Oct 30 20:28:52.622602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:52.622566 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" event={"ID":"32ef9ec4-a498-4de2-a9ac-368e26e5f67e","Type":"ContainerStarted","Data":"60e1ffa6ba8ecc349ab5fb30f5f09c69490b366a94daa9fe063aab7f7fdf8019"} Oct 30 20:28:52.623034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:52.622621 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" event={"ID":"32ef9ec4-a498-4de2-a9ac-368e26e5f67e","Type":"ContainerStarted","Data":"053e212820b2c5df5e2fe65182784af985099e12afa56905d0667c16148806c3"} Oct 30 20:28:52.623034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:52.622710 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:28:52.665923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:52.665859 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" podStartSLOduration=1.665839085 podStartE2EDuration="1.665839085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:52.664577396 +0000 UTC m=+3200.114070118" watchObservedRunningTime="2026-10-30 20:28:52.665839085 +0000 UTC m=+3200.115331812" Oct 30 20:28:53.056470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.056351 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 20:28:53.056745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.056721 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" podUID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" containerName="maas-api" containerID="cri-o://d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e" gracePeriod=30 Oct 30 20:28:53.346717 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.346685 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 20:28:53.439304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.439262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\") pod \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " Oct 30 20:28:53.439528 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.439429 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") pod \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " Oct 30 20:28:53.439635 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.439616 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\") pod \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " Oct 30 20:28:53.439688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.439673 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\") pod \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\" (UID: \"c3bc0c24-5f35-4f75-b22d-728604f7e6bb\") " Oct 30 20:28:53.440028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.439990 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca" pod "c3bc0c24-5f35-4f75-b22d-728604f7e6bb" (UID: "c3bc0c24-5f35-4f75-b22d-728604f7e6bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:53.440198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.440176 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:53.442032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.441999 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls" pod "c3bc0c24-5f35-4f75-b22d-728604f7e6bb" (UID: "c3bc0c24-5f35-4f75-b22d-728604f7e6bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:53.442032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.442017 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs" pod "c3bc0c24-5f35-4f75-b22d-728604f7e6bb" (UID: "c3bc0c24-5f35-4f75-b22d-728604f7e6bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:53.442032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.442025 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m" pod "c3bc0c24-5f35-4f75-b22d-728604f7e6bb" (UID: "c3bc0c24-5f35-4f75-b22d-728604f7e6bb"). InnerVolumeSpecName "kube-api-access-2944m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:53.542157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.542070 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-kube-api-access-2944m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:53.542157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.542114 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:53.542157 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.542127 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bc0c24-5f35-4f75-b22d-728604f7e6bb-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:28:53.634310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.634206 5991 generic.go:356] "Generic (PLEG): container finished" podID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" containerID="d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e" exitCode=0 Oct 30 20:28:53.634310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.634252 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" Oct 30 20:28:53.634881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.634299 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" event={"ID":"c3bc0c24-5f35-4f75-b22d-728604f7e6bb","Type":"ContainerDied","Data":"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e"} Oct 30 20:28:53.634881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.634396 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl" event={"ID":"c3bc0c24-5f35-4f75-b22d-728604f7e6bb","Type":"ContainerDied","Data":"d4eb48e9cb8da39e66bba84a949ddef19808a99d00038ed90ee647084b8fb8fc"} Oct 30 20:28:53.634881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.634422 5991 scope.go:122] "RemoveContainer" containerID="d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e" Oct 30 20:28:53.646468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.646442 5991 scope.go:122] "RemoveContainer" containerID="d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e" Oct 30 20:28:53.646776 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:28:53.646752 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e\": container with ID starting with d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e not found: ID does not exist" containerID="d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e" Oct 30 20:28:53.646864 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.646786 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e"} err="failed to get container status \"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e\": rpc error: code = NotFound desc = could not find container \"d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e\": container with ID starting with d28f23da6dc53f2a68b401756d1292e6ac18a6b5a1dd7602bd1550259383668e not found: ID does not exist" Oct 30 20:28:53.686747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.686714 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 20:28:53.699326 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:53.699283 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a165e5-57dc697d99-zldtl"] Oct 30 20:28:54.679480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:28:54.679443 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" path="/var/lib/kubelet/pods/c3bc0c24-5f35-4f75-b22d-728604f7e6bb/volumes" Oct 30 20:29:00.139068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:00.139029 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:29:03.637137 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:03.637105 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:29:10.205018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.204959 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 20:29:10.205582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.205262 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" podUID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" containerName="maas-api" containerID="cri-o://88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee" gracePeriod=30 Oct 30 20:29:10.484216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.484183 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 20:29:10.655221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655179 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\" (UniqueName: \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\") pod \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " Oct 30 20:29:10.655221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655226 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\") pod \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " Oct 30 20:29:10.655550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655346 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\") pod \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " Oct 30 20:29:10.655550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655514 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\") pod \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\" (UID: \"a1e5be3f-9f14-4ddb-b422-545ad1a62814\") " Oct 30 20:29:10.655738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655705 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca" pod "a1e5be3f-9f14-4ddb-b422-545ad1a62814" (UID: "a1e5be3f-9f14-4ddb-b422-545ad1a62814"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:10.656014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.655992 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5be3f-9f14-4ddb-b422-545ad1a62814-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:10.657509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.657481 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb" pod "a1e5be3f-9f14-4ddb-b422-545ad1a62814" (UID: "a1e5be3f-9f14-4ddb-b422-545ad1a62814"). InnerVolumeSpecName "kube-api-access-46flb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:10.657607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.657527 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls" pod "a1e5be3f-9f14-4ddb-b422-545ad1a62814" (UID: "a1e5be3f-9f14-4ddb-b422-545ad1a62814"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:10.657607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.657561 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs" pod "a1e5be3f-9f14-4ddb-b422-545ad1a62814" (UID: "a1e5be3f-9f14-4ddb-b422-545ad1a62814"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:10.756488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.756451 5991 generic.go:356] "Generic (PLEG): container finished" podID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" containerID="88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee" exitCode=0 Oct 30 20:29:10.756711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.756525 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" Oct 30 20:29:10.756711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.756697 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" event={"ID":"a1e5be3f-9f14-4ddb-b422-545ad1a62814","Type":"ContainerDied","Data":"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee"} Oct 30 20:29:10.756837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.756747 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5" event={"ID":"a1e5be3f-9f14-4ddb-b422-545ad1a62814","Type":"ContainerDied","Data":"b5b916a03e7380e8ea0e53599238201086fa050b9774552a005f973e260be10a"} Oct 30 20:29:10.756837 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.756781 5991 scope.go:122] "RemoveContainer" containerID="88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee" Oct 30 20:29:10.757874 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.757850 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46flb\" (UniqueName: \"kubernetes.io/projected/a1e5be3f-9f14-4ddb-b422-545ad1a62814-kube-api-access-46flb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:10.758004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.757893 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:10.758004 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.757910 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e5be3f-9f14-4ddb-b422-545ad1a62814-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:10.780006 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.779910 5991 scope.go:122] "RemoveContainer" containerID="88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee" Oct 30 20:29:10.780417 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:10.780339 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee\": container with ID starting with 88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee not found: ID does not exist" containerID="88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee" Oct 30 20:29:10.780499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.780435 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee"} err="failed to get container status \"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee\": rpc error: code = NotFound desc = could not find container \"88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee\": container with ID starting with 88ffc37fbebbf083d83fb51ba4192a38869603433c3290820a6ed72990abc7ee not found: ID does not exist" Oct 30 20:29:10.811683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.811648 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 20:29:10.823400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:10.823340 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c8fa69-d76fb7568-4vnv5"] Oct 30 20:29:12.015933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:12.015890 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 20:29:12.047090 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:12.047054 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 20:29:12.681625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:12.681582 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" path="/var/lib/kubelet/pods/a1e5be3f-9f14-4ddb-b422-545ad1a62814/volumes" Oct 30 20:29:17.016921 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:17.016872 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" containerID="cri-o://8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7" gracePeriod=25 Oct 30 20:29:17.047766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:17.047697 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" containerID="cri-o://9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3" gracePeriod=25 Oct 30 20:29:20.870161 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:20.870119 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:29:22.690201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:22.690167 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:22.690201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:22.690184 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:29.215325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:29.215287 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 20:29:29.245454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:29.245417 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 20:29:31.110344 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.110304 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 20:29:31.110858 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.110577 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" podUID="10d5e21e-b639-44fd-b15f-fb410753e91e" containerName="maas-api" containerID="cri-o://0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4" gracePeriod=30 Oct 30 20:29:31.374794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.374683 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 20:29:31.513597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.513558 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\" (UniqueName: \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\") pod \"10d5e21e-b639-44fd-b15f-fb410753e91e\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " Oct 30 20:29:31.513805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.513640 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\") pod \"10d5e21e-b639-44fd-b15f-fb410753e91e\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " Oct 30 20:29:31.513805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.513799 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") pod \"10d5e21e-b639-44fd-b15f-fb410753e91e\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " Oct 30 20:29:31.513933 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.513877 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\") pod \"10d5e21e-b639-44fd-b15f-fb410753e91e\" (UID: \"10d5e21e-b639-44fd-b15f-fb410753e91e\") " Oct 30 20:29:31.514301 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.514245 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca" pod "10d5e21e-b639-44fd-b15f-fb410753e91e" (UID: "10d5e21e-b639-44fd-b15f-fb410753e91e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:31.514462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.514340 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d5e21e-b639-44fd-b15f-fb410753e91e-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:31.515910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.515883 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8" pod "10d5e21e-b639-44fd-b15f-fb410753e91e" (UID: "10d5e21e-b639-44fd-b15f-fb410753e91e"). InnerVolumeSpecName "kube-api-access-bdnn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:31.516009 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.515888 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs" pod "10d5e21e-b639-44fd-b15f-fb410753e91e" (UID: "10d5e21e-b639-44fd-b15f-fb410753e91e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:31.516009 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.515919 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls" pod "10d5e21e-b639-44fd-b15f-fb410753e91e" (UID: "10d5e21e-b639-44fd-b15f-fb410753e91e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:31.615728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.615690 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdnn8\" (UniqueName: \"kubernetes.io/projected/10d5e21e-b639-44fd-b15f-fb410753e91e-kube-api-access-bdnn8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:31.615728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.615721 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:31.615728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.615731 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d5e21e-b639-44fd-b15f-fb410753e91e-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:31.901853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.901816 5991 generic.go:356] "Generic (PLEG): container finished" podID="10d5e21e-b639-44fd-b15f-fb410753e91e" containerID="0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4" exitCode=0 Oct 30 20:29:31.902012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.901873 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" Oct 30 20:29:31.902012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.901884 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" event={"ID":"10d5e21e-b639-44fd-b15f-fb410753e91e","Type":"ContainerDied","Data":"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4"} Oct 30 20:29:31.902012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.901937 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb" event={"ID":"10d5e21e-b639-44fd-b15f-fb410753e91e","Type":"ContainerDied","Data":"4d4f3dbfc9e7430996c7af6dd1ba1779124e1a884b98f8e077a87a129effe74e"} Oct 30 20:29:31.902012 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.901965 5991 scope.go:122] "RemoveContainer" containerID="0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4" Oct 30 20:29:31.913129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.913105 5991 scope.go:122] "RemoveContainer" containerID="0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4" Oct 30 20:29:31.913448 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:31.913417 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4\": container with ID starting with 0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4 not found: ID does not exist" containerID="0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4" Oct 30 20:29:31.913550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.913456 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4"} err="failed to get container status \"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4\": rpc error: code = NotFound desc = could not find container \"0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4\": container with ID starting with 0f5f89a5519be10d0241488fb5a2db931d4f047793fcb10f7d4b146958b7d0b4 not found: ID does not exist" Oct 30 20:29:31.960648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.960612 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 20:29:31.973259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:31.973225 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87ff9f-85fb64ccb9-5x4nb"] Oct 30 20:29:32.680424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:32.680394 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d5e21e-b639-44fd-b15f-fb410753e91e" path="/var/lib/kubelet/pods/10d5e21e-b639-44fd-b15f-fb410753e91e/volumes" Oct 30 20:29:32.690055 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:32.690025 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:32.690208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:32.690034 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:34.216480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:34.216423 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" containerID="cri-o://cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca" gracePeriod=25 Oct 30 20:29:34.246186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:34.246134 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" containerID="cri-o://805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac" gracePeriod=25 Oct 30 20:29:37.334975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.334944 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 20:29:37.369670 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.369642 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 20:29:37.396795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.396765 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:29:37.396990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.396966 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" podUID="df59d3d3-c326-422e-8033-600514a97a46" containerName="payload-processing" containerID="cri-o://0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de" gracePeriod=30 Oct 30 20:29:37.486976 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.486939 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\") pod \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " Oct 30 20:29:37.487189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487052 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\" (UniqueName: \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\") pod \"f684b144-bb7d-49fa-a3d7-fac34fb54598\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " Oct 30 20:29:37.487189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487173 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\" (UniqueName: \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\") pod \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\" (UID: \"d90b1645-a10d-475c-afa8-3c6c79c58cd2\") " Oct 30 20:29:37.487315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487236 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\") pod \"f684b144-bb7d-49fa-a3d7-fac34fb54598\" (UID: \"f684b144-bb7d-49fa-a3d7-fac34fb54598\") " Oct 30 20:29:37.487491 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487431 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume" pod "d90b1645-a10d-475c-afa8-3c6c79c58cd2" (UID: "d90b1645-a10d-475c-afa8-3c6c79c58cd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:37.487878 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487841 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b1645-a10d-475c-afa8-3c6c79c58cd2-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.487878 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.487850 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume" pod "f684b144-bb7d-49fa-a3d7-fac34fb54598" (UID: "f684b144-bb7d-49fa-a3d7-fac34fb54598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:37.490346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.490317 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv" pod "d90b1645-a10d-475c-afa8-3c6c79c58cd2" (UID: "d90b1645-a10d-475c-afa8-3c6c79c58cd2"). InnerVolumeSpecName "kube-api-access-x28pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:37.490493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.490416 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9" pod "f684b144-bb7d-49fa-a3d7-fac34fb54598" (UID: "f684b144-bb7d-49fa-a3d7-fac34fb54598"). InnerVolumeSpecName "kube-api-access-hq9f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:37.589218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.589182 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28pv\" (UniqueName: \"kubernetes.io/projected/d90b1645-a10d-475c-afa8-3c6c79c58cd2-kube-api-access-x28pv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.589218 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.589218 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f684b144-bb7d-49fa-a3d7-fac34fb54598-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.589486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.589231 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq9f9\" (UniqueName: \"kubernetes.io/projected/f684b144-bb7d-49fa-a3d7-fac34fb54598-kube-api-access-hq9f9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.648790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.648763 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:29:37.791924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.791892 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\" (UniqueName: \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\") pod \"df59d3d3-c326-422e-8033-600514a97a46\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " Oct 30 20:29:37.792145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.792000 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\") pod \"df59d3d3-c326-422e-8033-600514a97a46\" (UID: \"df59d3d3-c326-422e-8033-600514a97a46\") " Oct 30 20:29:37.792764 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.792731 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume" pod "df59d3d3-c326-422e-8033-600514a97a46" (UID: "df59d3d3-c326-422e-8033-600514a97a46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:37.794708 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.794678 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp" pod "df59d3d3-c326-422e-8033-600514a97a46" (UID: "df59d3d3-c326-422e-8033-600514a97a46"). InnerVolumeSpecName "kube-api-access-kdjrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:37.893971 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.893892 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjrp\" (UniqueName: \"kubernetes.io/projected/df59d3d3-c326-422e-8033-600514a97a46-kube-api-access-kdjrp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.893971 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.893933 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59d3d3-c326-422e-8033-600514a97a46-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.946844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.946806 5991 generic.go:356] "Generic (PLEG): container finished" podID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerID="9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3" exitCode=0 Oct 30 20:29:37.947057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.946861 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" Oct 30 20:29:37.947057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.946887 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" event={"ID":"d90b1645-a10d-475c-afa8-3c6c79c58cd2","Type":"ContainerDied","Data":"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3"} Oct 30 20:29:37.947057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.946942 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr" event={"ID":"d90b1645-a10d-475c-afa8-3c6c79c58cd2","Type":"ContainerDied","Data":"8cabc27d7be36ed9a86d7e347068c568a3a89eccb040e6494ac314b749dd41d6"} Oct 30 20:29:37.947057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.946970 5991 scope.go:122] "RemoveContainer" containerID="9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3" Oct 30 20:29:37.948552 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.948526 5991 generic.go:356] "Generic (PLEG): container finished" podID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerID="8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7" exitCode=0 Oct 30 20:29:37.948689 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.948603 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" event={"ID":"f684b144-bb7d-49fa-a3d7-fac34fb54598","Type":"ContainerDied","Data":"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7"} Oct 30 20:29:37.948689 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.948634 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" event={"ID":"f684b144-bb7d-49fa-a3d7-fac34fb54598","Type":"ContainerDied","Data":"bc23ef5d5cebb9024e0ce7e49aa6ef5ee9322f2107f9a4d92203490cc53d4555"} Oct 30 20:29:37.948818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.948696 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8" Oct 30 20:29:37.952163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.952139 5991 generic.go:356] "Generic (PLEG): container finished" podID="df59d3d3-c326-422e-8033-600514a97a46" containerID="0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de" exitCode=0 Oct 30 20:29:37.952310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.952181 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" event={"ID":"df59d3d3-c326-422e-8033-600514a97a46","Type":"ContainerDied","Data":"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de"} Oct 30 20:29:37.952310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.952186 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" Oct 30 20:29:37.952310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.952217 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2" event={"ID":"df59d3d3-c326-422e-8033-600514a97a46","Type":"ContainerDied","Data":"c8eb7345ea553828eb76c3017384ffe88a28f4061d21a5f9817be65e7aa5158e"} Oct 30 20:29:37.961348 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.961321 5991 scope.go:122] "RemoveContainer" containerID="9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3" Oct 30 20:29:37.961729 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:37.961694 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3\": container with ID starting with 9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3 not found: ID does not exist" containerID="9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3" Oct 30 20:29:37.961853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.961747 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3"} err="failed to get container status \"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3\": rpc error: code = NotFound desc = could not find container \"9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3\": container with ID starting with 9046128774f30eb4979518a07db5528b772a0e667a12bcc26599dc9331172ef3 not found: ID does not exist" Oct 30 20:29:37.961853 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.961774 5991 scope.go:122] "RemoveContainer" containerID="8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7" Oct 30 20:29:37.975227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.975199 5991 scope.go:122] "RemoveContainer" containerID="8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7" Oct 30 20:29:37.975569 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:37.975548 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7\": container with ID starting with 8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7 not found: ID does not exist" containerID="8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7" Oct 30 20:29:37.975625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.975578 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7"} err="failed to get container status \"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7\": rpc error: code = NotFound desc = could not find container \"8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7\": container with ID starting with 8e594233b258b0954a2281ecb8a90acbc3a15db5edaa5c41f11530b16ee501e7 not found: ID does not exist" Oct 30 20:29:37.975625 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.975598 5991 scope.go:122] "RemoveContainer" containerID="0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de" Oct 30 20:29:37.984929 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.984909 5991 scope.go:122] "RemoveContainer" containerID="0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de" Oct 30 20:29:37.985208 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:37.985188 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de\": container with ID starting with 0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de not found: ID does not exist" containerID="0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de" Oct 30 20:29:37.985266 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:37.985216 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de"} err="failed to get container status \"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de\": rpc error: code = NotFound desc = could not find container \"0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de\": container with ID starting with 0eb8a200330f1b708daca7c683731cf668197c1541a02f5e04191d85b011f2de not found: ID does not exist" Oct 30 20:29:38.034546 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.034513 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 20:29:38.046728 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.046699 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87ff9f-586557c86-w5rb8"] Oct 30 20:29:38.083308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.083275 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 20:29:38.086451 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.086417 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87ff9f-6bb599fc9-d6hdr"] Oct 30 20:29:38.136611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.136576 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:29:38.143873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.143843 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c19563-6f7b4cccbd-t5cj2"] Oct 30 20:29:38.523820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.523781 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:38.524214 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.523940 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:38.678623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.678588 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" path="/var/lib/kubelet/pods/d90b1645-a10d-475c-afa8-3c6c79c58cd2/volumes" Oct 30 20:29:38.679138 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.679123 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df59d3d3-c326-422e-8033-600514a97a46" path="/var/lib/kubelet/pods/df59d3d3-c326-422e-8033-600514a97a46/volumes" Oct 30 20:29:38.679500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:38.679487 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" path="/var/lib/kubelet/pods/f684b144-bb7d-49fa-a3d7-fac34fb54598/volumes" Oct 30 20:29:43.451461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.451426 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:29:43.733318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.733279 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 20:29:43.733628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.733595 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" podUID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" containerName="maas-api" containerID="cri-o://be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d" gracePeriod=30 Oct 30 20:29:43.985702 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.985623 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 20:29:43.995183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.995151 5991 generic.go:356] "Generic (PLEG): container finished" podID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" containerID="be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d" exitCode=0 Oct 30 20:29:43.995337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.995195 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" Oct 30 20:29:43.995337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.995242 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" event={"ID":"b98ff840-e3a2-44e3-9e14-3881d883dbcb","Type":"ContainerDied","Data":"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d"} Oct 30 20:29:43.995337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.995295 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9" event={"ID":"b98ff840-e3a2-44e3-9e14-3881d883dbcb","Type":"ContainerDied","Data":"5f3b724f7834643711ee8b6c2d9bfe844a9dd2aac2a7048b1fafd22cd5187861"} Oct 30 20:29:43.995337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:43.995323 5991 scope.go:122] "RemoveContainer" containerID="be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d" Oct 30 20:29:44.005259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.005237 5991 scope.go:122] "RemoveContainer" containerID="be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d" Oct 30 20:29:44.005598 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:44.005569 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d\": container with ID starting with be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d not found: ID does not exist" containerID="be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d" Oct 30 20:29:44.005656 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.005608 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d"} err="failed to get container status \"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d\": rpc error: code = NotFound desc = could not find container \"be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d\": container with ID starting with be08497c34329d809e7f5d8a3c8a3cb693ad351d0f20ce8f881768d74abc5d9d not found: ID does not exist" Oct 30 20:29:44.183103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183067 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\") pod \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " Oct 30 20:29:44.183103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183103 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\" (UniqueName: \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\") pod \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " Oct 30 20:29:44.183351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183203 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\") pod \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " Oct 30 20:29:44.183351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183347 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\") pod \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\" (UID: \"b98ff840-e3a2-44e3-9e14-3881d883dbcb\") " Oct 30 20:29:44.183567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183446 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca" pod "b98ff840-e3a2-44e3-9e14-3881d883dbcb" (UID: "b98ff840-e3a2-44e3-9e14-3881d883dbcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:44.183851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.183828 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ff840-e3a2-44e3-9e14-3881d883dbcb-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:44.185297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.185267 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls" pod "b98ff840-e3a2-44e3-9e14-3881d883dbcb" (UID: "b98ff840-e3a2-44e3-9e14-3881d883dbcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:44.185413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.185317 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs" pod "b98ff840-e3a2-44e3-9e14-3881d883dbcb" (UID: "b98ff840-e3a2-44e3-9e14-3881d883dbcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:44.185413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.185378 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj" pod "b98ff840-e3a2-44e3-9e14-3881d883dbcb" (UID: "b98ff840-e3a2-44e3-9e14-3881d883dbcb"). InnerVolumeSpecName "kube-api-access-bcctj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:44.285136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.285035 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcctj\" (UniqueName: \"kubernetes.io/projected/b98ff840-e3a2-44e3-9e14-3881d883dbcb-kube-api-access-bcctj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:44.285136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.285071 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:44.285136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.285085 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ff840-e3a2-44e3-9e14-3881d883dbcb-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:44.347130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.347093 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 20:29:44.359357 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.359324 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-21c4ab-798b8cd999-7fnp9"] Oct 30 20:29:44.680192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:44.680102 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" path="/var/lib/kubelet/pods/b98ff840-e3a2-44e3-9e14-3881d883dbcb/volumes" Oct 30 20:29:45.246940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:45.246907 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 20:29:48.524227 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:48.524179 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:48.524747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:48.524187 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:50.633090 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:50.633052 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 20:29:50.868605 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:50.868563 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:29:52.743639 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:52.743602 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 20:29:52.744070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:52.743835 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" containerID="cri-o://39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064" gracePeriod=30 Oct 30 20:29:54.520250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.520223 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 20:29:54.553400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.549637 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 20:29:54.627249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627202 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\" (UniqueName: \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\") pod \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " Oct 30 20:29:54.627249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627247 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\") pod \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\" (UID: \"d7754cee-5ad8-4cc3-8fbd-751ecb155b51\") " Oct 30 20:29:54.627579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627309 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\") pod \"137b2e28-6d44-4676-81fd-5918945a4f1e\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " Oct 30 20:29:54.627579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627417 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\" (UniqueName: \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\") pod \"137b2e28-6d44-4676-81fd-5918945a4f1e\" (UID: \"137b2e28-6d44-4676-81fd-5918945a4f1e\") " Oct 30 20:29:54.627811 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627763 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume" pod "137b2e28-6d44-4676-81fd-5918945a4f1e" (UID: "137b2e28-6d44-4676-81fd-5918945a4f1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:54.627888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627770 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume" pod "d7754cee-5ad8-4cc3-8fbd-751ecb155b51" (UID: "d7754cee-5ad8-4cc3-8fbd-751ecb155b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:54.627983 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627961 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:54.628043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.627988 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b2e28-6d44-4676-81fd-5918945a4f1e-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:54.629599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.629569 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c" pod "137b2e28-6d44-4676-81fd-5918945a4f1e" (UID: "137b2e28-6d44-4676-81fd-5918945a4f1e"). InnerVolumeSpecName "kube-api-access-mb82c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:54.629726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.629633 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w" pod "d7754cee-5ad8-4cc3-8fbd-751ecb155b51" (UID: "d7754cee-5ad8-4cc3-8fbd-751ecb155b51"). InnerVolumeSpecName "kube-api-access-qbt7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:54.729684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.729641 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbt7w\" (UniqueName: \"kubernetes.io/projected/d7754cee-5ad8-4cc3-8fbd-751ecb155b51-kube-api-access-qbt7w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:54.729684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:54.729680 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb82c\" (UniqueName: \"kubernetes.io/projected/137b2e28-6d44-4676-81fd-5918945a4f1e-kube-api-access-mb82c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:55.069585 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.069550 5991 generic.go:356] "Generic (PLEG): container finished" podID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerID="cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca" exitCode=0 Oct 30 20:29:55.069806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.069623 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" Oct 30 20:29:55.069806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.069632 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" event={"ID":"d7754cee-5ad8-4cc3-8fbd-751ecb155b51","Type":"ContainerDied","Data":"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca"} Oct 30 20:29:55.069806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.069688 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6" event={"ID":"d7754cee-5ad8-4cc3-8fbd-751ecb155b51","Type":"ContainerDied","Data":"5ebaf62b24283bca50304839671211b575edf1213e1f2dfc0cc80fcddeff0c97"} Oct 30 20:29:55.069806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.069712 5991 scope.go:122] "RemoveContainer" containerID="cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca" Oct 30 20:29:55.071539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.071512 5991 generic.go:356] "Generic (PLEG): container finished" podID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerID="805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac" exitCode=0 Oct 30 20:29:55.071684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.071589 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" Oct 30 20:29:55.071684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.071587 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" event={"ID":"137b2e28-6d44-4676-81fd-5918945a4f1e","Type":"ContainerDied","Data":"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac"} Oct 30 20:29:55.071684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.071659 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn" event={"ID":"137b2e28-6d44-4676-81fd-5918945a4f1e","Type":"ContainerDied","Data":"9de15b3eacc4e6b60350d0dc3aa8501231dc24c63b56b0095fa9c034d9475756"} Oct 30 20:29:55.080739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.080715 5991 scope.go:122] "RemoveContainer" containerID="cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca" Oct 30 20:29:55.081127 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:55.081105 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca\": container with ID starting with cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca not found: ID does not exist" containerID="cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca" Oct 30 20:29:55.081189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.081139 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca"} err="failed to get container status \"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca\": rpc error: code = NotFound desc = could not find container \"cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca\": container with ID starting with cd28cacc7b244be18d8eb5a3eefa82aeef374545f8decc704033a7c071ad8aca not found: ID does not exist" Oct 30 20:29:55.081189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.081159 5991 scope.go:122] "RemoveContainer" containerID="805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac" Oct 30 20:29:55.091129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.091100 5991 scope.go:122] "RemoveContainer" containerID="805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac" Oct 30 20:29:55.091425 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:55.091399 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac\": container with ID starting with 805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac not found: ID does not exist" containerID="805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac" Oct 30 20:29:55.091485 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.091436 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac"} err="failed to get container status \"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac\": rpc error: code = NotFound desc = could not find container \"805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac\": container with ID starting with 805a075f90b4b2ca4ee01c9c1fc270ea2457089ae033ecc2132516a8d5caddac not found: ID does not exist" Oct 30 20:29:55.125968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.125932 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 20:29:55.137260 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.137230 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-21c4ab-7d6fb6f887-zntnn"] Oct 30 20:29:55.174791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.174757 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 20:29:55.178399 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.178354 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-21c4ab-58967cd46-mpnr6"] Oct 30 20:29:55.633812 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.633766 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" containerID="cri-o://12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae" gracePeriod=25 Oct 30 20:29:55.705958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:55.705916 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:56.681028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:56.680989 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" path="/var/lib/kubelet/pods/137b2e28-6d44-4676-81fd-5918945a4f1e/volumes" Oct 30 20:29:56.681663 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:56.681640 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" path="/var/lib/kubelet/pods/d7754cee-5ad8-4cc3-8fbd-751ecb155b51/volumes" Oct 30 20:29:58.028957 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.028886 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 20:29:58.098317 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.098282 5991 generic.go:356] "Generic (PLEG): container finished" podID="a8debdeb-b068-458e-866b-4246249c3b52" containerID="39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064" exitCode=0 Oct 30 20:29:58.098524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.098345 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" Oct 30 20:29:58.098524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.098381 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerDied","Data":"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064"} Oct 30 20:29:58.098524 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.098493 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk" event={"ID":"a8debdeb-b068-458e-866b-4246249c3b52","Type":"ContainerDied","Data":"2a966f8ecd2e4bc2bd956d5a3ab75cbda62f7f1eee4b11eb12dd279e78a4ad4c"} Oct 30 20:29:58.098706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.098527 5991 scope.go:122] "RemoveContainer" containerID="39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064" Oct 30 20:29:58.115068 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.115040 5991 scope.go:122] "RemoveContainer" containerID="f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087" Oct 30 20:29:58.127854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.127822 5991 scope.go:122] "RemoveContainer" containerID="39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064" Oct 30 20:29:58.128227 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:58.128197 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064\": container with ID starting with 39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064 not found: ID does not exist" containerID="39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064" Oct 30 20:29:58.128337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.128240 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064"} err="failed to get container status \"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064\": rpc error: code = NotFound desc = could not find container \"39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064\": container with ID starting with 39bd42c79216c05c0e03ada41d0c816f7d06138cc920b2374267aa9791e7b064 not found: ID does not exist" Oct 30 20:29:58.128337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.128266 5991 scope.go:122] "RemoveContainer" containerID="f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087" Oct 30 20:29:58.128674 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:29:58.128650 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087\": container with ID starting with f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087 not found: ID does not exist" containerID="f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087" Oct 30 20:29:58.128721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.128684 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087"} err="failed to get container status \"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087\": rpc error: code = NotFound desc = could not find container \"f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087\": container with ID starting with f4a046862d62da583e90c41624b498dd608900cc8409b69ae6799a20c456f087 not found: ID does not exist" Oct 30 20:29:58.173191 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173146 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173238 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173294 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173332 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173432 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173418 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173452 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173462 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173524 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173585 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173622 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\") pod \"a8debdeb-b068-458e-866b-4246249c3b52\" (UID: \"a8debdeb-b068-458e-866b-4246249c3b52\") " Oct 30 20:29:58.173657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173635 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:58.173989 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173861 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:58.174044 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.173991 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:58.174132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.174113 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.174187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.174137 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.174187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.174151 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.174187 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.174163 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8debdeb-b068-458e-866b-4246249c3b52-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.174341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.174218 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:58.176006 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.175974 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:58.176145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.176108 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:58.176209 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.176167 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "kube-api-access-mnb5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:58.176407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.176388 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo" pod "a8debdeb-b068-458e-866b-4246249c3b52" (UID: "a8debdeb-b068-458e-866b-4246249c3b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:29:58.205402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.205353 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 20:29:58.276061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.276018 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.276061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.276054 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8debdeb-b068-458e-866b-4246249c3b52-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.276061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.276063 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.276061 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.276071 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8debdeb-b068-458e-866b-4246249c3b52-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.276387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.276080 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb5m\" (UniqueName: \"kubernetes.io/projected/a8debdeb-b068-458e-866b-4246249c3b52-kube-api-access-mnb5m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:29:58.448886 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.448843 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 20:29:58.456186 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.456151 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-21c4ab-openshift-default-7b86f4bf57-zlqkk"] Oct 30 20:29:58.680468 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:29:58.680385 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8debdeb-b068-458e-866b-4246249c3b52" path="/var/lib/kubelet/pods/a8debdeb-b068-458e-866b-4246249c3b52/volumes" Oct 30 20:30:00.219988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.219947 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:00.220610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220593 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" containerName="maas-api" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220612 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" containerName="maas-api" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220623 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220628 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220643 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220648 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220665 5991 cpu_manager.go:414] "containerMap: removing container" podUID="df59d3d3-c326-422e-8033-600514a97a46" containerName="payload-processing" Oct 30 20:30:00.220667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220670 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df59d3d3-c326-422e-8033-600514a97a46" containerName="payload-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220688 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220694 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220701 5991 cpu_manager.go:414] "containerMap: removing container" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220706 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220718 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220723 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220731 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220735 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220745 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220750 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220758 5991 cpu_manager.go:414] "containerMap: removing container" podUID="10d5e21e-b639-44fd-b15f-fb410753e91e" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220763 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d5e21e-b639-44fd-b15f-fb410753e91e" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220778 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220783 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220864 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e5be3f-9f14-4ddb-b422-545ad1a62814" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220874 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="137b2e28-6d44-4676-81fd-5918945a4f1e" containerName="payload-pre-processing" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220882 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220889 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bc0c24-5f35-4f75-b22d-728604f7e6bb" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220895 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8debdeb-b068-458e-866b-4246249c3b52" containerName="istio-proxy" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220901 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d5e21e-b639-44fd-b15f-fb410753e91e" containerName="maas-api" Oct 30 20:30:00.220904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220911 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7754cee-5ad8-4cc3-8fbd-751ecb155b51" containerName="payload-processing" Oct 30 20:30:00.221681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220917 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98ff840-e3a2-44e3-9e14-3881d883dbcb" containerName="maas-api" Oct 30 20:30:00.221681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220922 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90b1645-a10d-475c-afa8-3c6c79c58cd2" containerName="payload-pre-processing" Oct 30 20:30:00.221681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220928 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f684b144-bb7d-49fa-a3d7-fac34fb54598" containerName="payload-processing" Oct 30 20:30:00.221681 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.220934 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="df59d3d3-c326-422e-8033-600514a97a46" containerName="payload-processing" Oct 30 20:30:00.226395 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.226345 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.246856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.246822 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:00.268048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.267995 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" containerID="cri-o://dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b" gracePeriod=45 Oct 30 20:30:00.279167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.279137 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:00.285039 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.284998 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.303105 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.303073 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:30:00.314421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.314335 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:00.400705 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.400664 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvt9\" (UniqueName: \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.400915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.400726 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.400986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.400916 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzs5\" (UniqueName: \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.400986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.400960 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.501920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.501877 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvt9\" (UniqueName: \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.502112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.501926 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.502112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.502033 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzs5\" (UniqueName: \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.502112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.502070 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.502685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.502656 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.502790 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.502773 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.523209 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.523171 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzs5\" (UniqueName: \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.523507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.523483 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvt9\" (UniqueName: \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.550086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.550058 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:00.551412 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.551387 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 20:30:00.662940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.662905 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:00.703923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.703883 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.704082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.703947 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\" (UniqueName: \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.704082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.704014 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.704161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.704138 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.704281 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.704264 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\" (UniqueName: \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.704396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.704357 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\") pod \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\" (UID: \"e67103ec-b3dd-4fc5-892b-1c1a9b113b07\") " Oct 30 20:30:00.705100 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.705067 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.705100 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.705089 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.705243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.705148 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.706793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.706764 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.706944 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.706919 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "kube-api-access-6c7nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:00.707306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.707284 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs" pod "e67103ec-b3dd-4fc5-892b-1c1a9b113b07" (UID: "e67103ec-b3dd-4fc5-892b-1c1a9b113b07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805819 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805852 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805860 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805867 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805878 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c7nf\" (UniqueName: \"kubernetes.io/projected/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-kube-api-access-6c7nf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.805907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.805887 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67103ec-b3dd-4fc5-892b-1c1a9b113b07-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.867291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.867258 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:00.868622 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:30:00.868592 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bea4eb_4c43_4dcb_bd45_48e9e0123a28.slice/crio-4405335e2dd22ad7b84af150572574b6ce1ade23d4d92770f68d0279fb365e11 WatchSource:0}: Error finding container 4405335e2dd22ad7b84af150572574b6ce1ade23d4d92770f68d0279fb365e11: Status 404 returned error can't find the container with id 4405335e2dd22ad7b84af150572574b6ce1ade23d4d92770f68d0279fb365e11 Oct 30 20:30:00.977461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:00.977433 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:00.978214 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:30:00.978190 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb156438c_2e22_43d0_accb_e534fa7843ed.slice/crio-f69fdf8d716058355b9aacf567d14ebd5273c605ddb621d7086eddcc06c0519d WatchSource:0}: Error finding container f69fdf8d716058355b9aacf567d14ebd5273c605ddb621d7086eddcc06c0519d: Status 404 returned error can't find the container with id f69fdf8d716058355b9aacf567d14ebd5273c605ddb621d7086eddcc06c0519d Oct 30 20:30:01.120353 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.120312 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerStarted","Data":"4405335e2dd22ad7b84af150572574b6ce1ade23d4d92770f68d0279fb365e11"} Oct 30 20:30:01.121641 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.121612 5991 generic.go:356] "Generic (PLEG): container finished" podID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerID="dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b" exitCode=0 Oct 30 20:30:01.121782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.121642 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" event={"ID":"e67103ec-b3dd-4fc5-892b-1c1a9b113b07","Type":"ContainerDied","Data":"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b"} Oct 30 20:30:01.121782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.121669 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" Oct 30 20:30:01.121782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.121679 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr" event={"ID":"e67103ec-b3dd-4fc5-892b-1c1a9b113b07","Type":"ContainerDied","Data":"0f6f2d900f42c2caf4b71c77ec8f778c73b43e81ea1c58078065638dd5be7c83"} Oct 30 20:30:01.121782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.121699 5991 scope.go:122] "RemoveContainer" containerID="dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b" Oct 30 20:30:01.123064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.123040 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerStarted","Data":"f69fdf8d716058355b9aacf567d14ebd5273c605ddb621d7086eddcc06c0519d"} Oct 30 20:30:01.132444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.132423 5991 scope.go:122] "RemoveContainer" containerID="dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b" Oct 30 20:30:01.132793 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:01.132771 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b\": container with ID starting with dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b not found: ID does not exist" containerID="dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b" Oct 30 20:30:01.132841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.132804 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b"} err="failed to get container status \"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b\": rpc error: code = NotFound desc = could not find container \"dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b\": container with ID starting with dead9679b406c1e1bd05ad092ef01a5faf74e5def9ad99ef6f995d152ea24a0b not found: ID does not exist" Oct 30 20:30:01.180820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.180784 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 20:30:01.186342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:01.186307 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-2-w2-284bf6-kserve-65cd8b57bd-lhjdr"] Oct 30 20:30:02.130785 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.130698 5991 generic.go:356] "Generic (PLEG): container finished" podID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerID="4557ca9e6d11a5708b186e85944333c5d137066df3b8421ea7d639bd27690162" exitCode=60 Oct 30 20:30:02.131264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.130778 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerDied","Data":"4557ca9e6d11a5708b186e85944333c5d137066df3b8421ea7d639bd27690162"} Oct 30 20:30:02.131264 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.131025 5991 scope.go:122] "RemoveContainer" containerID="4557ca9e6d11a5708b186e85944333c5d137066df3b8421ea7d639bd27690162" Oct 30 20:30:02.134036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.134005 5991 generic.go:356] "Generic (PLEG): container finished" podID="b156438c-2e22-43d0-accb-e534fa7843ed" containerID="4e3c15bf670435be8ac2d8c923a33452446bdc6b55a581b1483a54597856b107" exitCode=60 Oct 30 20:30:02.134036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.134032 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerDied","Data":"4e3c15bf670435be8ac2d8c923a33452446bdc6b55a581b1483a54597856b107"} Oct 30 20:30:02.134568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.134353 5991 scope.go:122] "RemoveContainer" containerID="4e3c15bf670435be8ac2d8c923a33452446bdc6b55a581b1483a54597856b107" Oct 30 20:30:02.680794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:02.680710 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" path="/var/lib/kubelet/pods/e67103ec-b3dd-4fc5-892b-1c1a9b113b07/volumes" Oct 30 20:30:03.143816 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.143773 5991 generic.go:356] "Generic (PLEG): container finished" podID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerID="1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c" exitCode=60 Oct 30 20:30:03.144258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.143854 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerDied","Data":"1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c"} Oct 30 20:30:03.144258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.143944 5991 scope.go:122] "RemoveContainer" containerID="4557ca9e6d11a5708b186e85944333c5d137066df3b8421ea7d639bd27690162" Oct 30 20:30:03.144430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.144262 5991 scope.go:122] "RemoveContainer" containerID="1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c" Oct 30 20:30:03.144656 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:03.144631 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p_odh-ai-gateway-infra(02bea4eb-4c43-4dcb-bd45-48e9e0123a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" Oct 30 20:30:03.146330 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.146306 5991 generic.go:356] "Generic (PLEG): container finished" podID="b156438c-2e22-43d0-accb-e534fa7843ed" containerID="969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5" exitCode=60 Oct 30 20:30:03.146475 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.146387 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerDied","Data":"969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5"} Oct 30 20:30:03.146680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.146663 5991 scope.go:122] "RemoveContainer" containerID="969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5" Oct 30 20:30:03.146957 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:03.146939 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm_odh-ai-gateway-infra(b156438c-2e22-43d0-accb-e534fa7843ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" Oct 30 20:30:03.157318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.157299 5991 scope.go:122] "RemoveContainer" containerID="4e3c15bf670435be8ac2d8c923a33452446bdc6b55a581b1483a54597856b107" Oct 30 20:30:03.656255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.656219 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 20:30:03.694825 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:03.694788 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 20:30:04.157091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:04.157057 5991 scope.go:122] "RemoveContainer" containerID="1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c" Oct 30 20:30:04.157594 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:04.157427 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p_odh-ai-gateway-infra(02bea4eb-4c43-4dcb-bd45-48e9e0123a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" Oct 30 20:30:04.158948 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:04.158930 5991 scope.go:122] "RemoveContainer" containerID="969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5" Oct 30 20:30:04.159220 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:04.159197 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm_odh-ai-gateway-infra(b156438c-2e22-43d0-accb-e534fa7843ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" Oct 30 20:30:05.705952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:05.705913 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:08.657477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:08.657403 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" containerID="cri-o://a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e" gracePeriod=25 Oct 30 20:30:08.695958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:08.695907 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" podUID="23a40f54-dfd4-41b8-86e5-045f359c2be5" containerName="payload-pre-processing" containerID="cri-o://8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7" gracePeriod=25 Oct 30 20:30:08.997593 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:08.997558 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 20:30:09.102932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.102899 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\") pod \"23a40f54-dfd4-41b8-86e5-045f359c2be5\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " Oct 30 20:30:09.103131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.102963 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\" (UniqueName: \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\") pod \"23a40f54-dfd4-41b8-86e5-045f359c2be5\" (UID: \"23a40f54-dfd4-41b8-86e5-045f359c2be5\") " Oct 30 20:30:09.103358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.103333 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume" pod "23a40f54-dfd4-41b8-86e5-045f359c2be5" (UID: "23a40f54-dfd4-41b8-86e5-045f359c2be5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:09.103655 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.103633 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a40f54-dfd4-41b8-86e5-045f359c2be5-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:09.105568 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.105541 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9" pod "23a40f54-dfd4-41b8-86e5-045f359c2be5" (UID: "23a40f54-dfd4-41b8-86e5-045f359c2be5"). InnerVolumeSpecName "kube-api-access-w7cn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:09.194611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.194572 5991 generic.go:356] "Generic (PLEG): container finished" podID="23a40f54-dfd4-41b8-86e5-045f359c2be5" containerID="8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7" exitCode=0 Oct 30 20:30:09.194611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.194605 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" event={"ID":"23a40f54-dfd4-41b8-86e5-045f359c2be5","Type":"ContainerDied","Data":"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7"} Oct 30 20:30:09.194872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.194631 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" Oct 30 20:30:09.194872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.194671 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc" event={"ID":"23a40f54-dfd4-41b8-86e5-045f359c2be5","Type":"ContainerDied","Data":"9f6e65c889647e2468b34ee273552bef70b373402b4dbf25bec3a7ec1a5b273b"} Oct 30 20:30:09.194872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.194701 5991 scope.go:122] "RemoveContainer" containerID="8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7" Oct 30 20:30:09.205182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.205138 5991 scope.go:122] "RemoveContainer" containerID="8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7" Oct 30 20:30:09.205557 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:09.205534 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7\": container with ID starting with 8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7 not found: ID does not exist" containerID="8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7" Oct 30 20:30:09.205664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.205557 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cn9\" (UniqueName: \"kubernetes.io/projected/23a40f54-dfd4-41b8-86e5-045f359c2be5-kube-api-access-w7cn9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:09.205664 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.205569 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7"} err="failed to get container status \"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7\": rpc error: code = NotFound desc = could not find container \"8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7\": container with ID starting with 8c249e8b74bad3e71a88d20a286e793a9d66656fbecf82b3576af0e5d8591be7 not found: ID does not exist" Oct 30 20:30:09.258069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.258029 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 20:30:09.270307 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:09.270267 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-01f7d3-5d9cf89dd7-hrnmc"] Oct 30 20:30:10.679335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:10.679300 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a40f54-dfd4-41b8-86e5-045f359c2be5" path="/var/lib/kubelet/pods/23a40f54-dfd4-41b8-86e5-045f359c2be5/volumes" Oct 30 20:30:11.774604 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:11.774568 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:30:11.775129 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:11.774824 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" podUID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" containerName="maas-api" containerID="cri-o://b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641" gracePeriod=30 Oct 30 20:30:11.986737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:11.981469 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:30:11.986737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:11.981779 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" podUID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" containerName="payload-processing" containerID="cri-o://60e1ffa6ba8ecc349ab5fb30f5f09c69490b366a94daa9fe063aab7f7fdf8019" gracePeriod=30 Oct 30 20:30:12.128146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.128108 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:30:12.221905 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.221865 5991 generic.go:356] "Generic (PLEG): container finished" podID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" containerID="60e1ffa6ba8ecc349ab5fb30f5f09c69490b366a94daa9fe063aab7f7fdf8019" exitCode=0 Oct 30 20:30:12.222119 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.221944 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" event={"ID":"32ef9ec4-a498-4de2-a9ac-368e26e5f67e","Type":"ContainerDied","Data":"60e1ffa6ba8ecc349ab5fb30f5f09c69490b366a94daa9fe063aab7f7fdf8019"} Oct 30 20:30:12.223727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.223697 5991 generic.go:356] "Generic (PLEG): container finished" podID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" containerID="b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641" exitCode=0 Oct 30 20:30:12.223843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.223737 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" event={"ID":"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f","Type":"ContainerDied","Data":"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641"} Oct 30 20:30:12.223843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.223773 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" event={"ID":"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f","Type":"ContainerDied","Data":"4e3b5c731efaefa10346eba12eff8dce5a1e226781c70779acf4256bf2a42d37"} Oct 30 20:30:12.223843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.223792 5991 scope.go:122] "RemoveContainer" containerID="b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641" Oct 30 20:30:12.223962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.223903 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2" Oct 30 20:30:12.245397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.245341 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\") pod \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " Oct 30 20:30:12.245618 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.245413 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\") pod \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " Oct 30 20:30:12.245687 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.245630 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\") pod \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " Oct 30 20:30:12.245923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.245894 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca" pod "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" (UID: "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:12.246339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.246047 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\" (UniqueName: \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\") pod \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\" (UID: \"c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f\") " Oct 30 20:30:12.246586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.246568 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.247609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.247580 5991 scope.go:122] "RemoveContainer" containerID="b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641" Oct 30 20:30:12.248325 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:12.248250 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641\": container with ID starting with b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641 not found: ID does not exist" containerID="b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641" Oct 30 20:30:12.248325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.248299 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641"} err="failed to get container status \"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641\": rpc error: code = NotFound desc = could not find container \"b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641\": container with ID starting with b61fb2d6516f8608b51b8ee1c77e7884a110e9de0d8f970df2b24177ef998641 not found: ID does not exist" Oct 30 20:30:12.248606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.248495 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs" pod "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" (UID: "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:12.250152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.250116 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq" pod "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" (UID: "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f"). InnerVolumeSpecName "kube-api-access-kphtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:12.250871 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.250830 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls" pod "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" (UID: "c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:12.264101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.264071 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:30:12.347902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.347813 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\" (UniqueName: \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\") pod \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " Oct 30 20:30:12.347902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.347877 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\") pod \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\" (UID: \"32ef9ec4-a498-4de2-a9ac-368e26e5f67e\") " Oct 30 20:30:12.348306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.348280 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume" pod "32ef9ec4-a498-4de2-a9ac-368e26e5f67e" (UID: "32ef9ec4-a498-4de2-a9ac-368e26e5f67e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:12.348409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.348337 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.348409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.348357 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kphtq\" (UniqueName: \"kubernetes.io/projected/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-kube-api-access-kphtq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.348409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.348387 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.350245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.350212 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4" pod "32ef9ec4-a498-4de2-a9ac-368e26e5f67e" (UID: "32ef9ec4-a498-4de2-a9ac-368e26e5f67e"). InnerVolumeSpecName "kube-api-access-r6jg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:12.356889 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.356851 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp"] Oct 30 20:30:12.357906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357876 5991 cpu_manager.go:414] "containerMap: removing container" podUID="23a40f54-dfd4-41b8-86e5-045f359c2be5" containerName="payload-pre-processing" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357906 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a40f54-dfd4-41b8-86e5-045f359c2be5" containerName="payload-pre-processing" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357959 5991 cpu_manager.go:414] "containerMap: removing container" podUID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" containerName="payload-processing" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357968 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" containerName="payload-processing" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357989 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.357996 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358009 5991 cpu_manager.go:414] "containerMap: removing container" podUID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" containerName="maas-api" Oct 30 20:30:12.358030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358017 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" containerName="maas-api" Oct 30 20:30:12.358401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358180 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a40f54-dfd4-41b8-86e5-045f359c2be5" containerName="payload-pre-processing" Oct 30 20:30:12.358401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358201 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" containerName="payload-processing" Oct 30 20:30:12.358401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358213 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67103ec-b3dd-4fc5-892b-1c1a9b113b07" containerName="main" Oct 30 20:30:12.358401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.358222 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" containerName="maas-api" Oct 30 20:30:12.362706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.362678 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.377749 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.377716 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:30:12.409327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.409279 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp"] Oct 30 20:30:12.449486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.449448 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.449695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.449517 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr88\" (UniqueName: \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.449695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.449622 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.449941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.449913 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jg4\" (UniqueName: \"kubernetes.io/projected/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-kube-api-access-r6jg4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.449987 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.449950 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ef9ec4-a498-4de2-a9ac-368e26e5f67e-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:12.551894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.551849 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.552108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.551910 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr88\" (UniqueName: \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.552108 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.551943 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.552943 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.552657 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.555500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.555472 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.572221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.572170 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:30:12.577604 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.577570 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr88\" (UniqueName: \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\") pod \"maas-api-revoke-sub-d6d9a785-bfhbp\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.577780 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.577760 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-843ea8-76764b9d54-pjbx2"] Oct 30 20:30:12.677428 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.677334 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:12.680442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.680406 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f" path="/var/lib/kubelet/pods/c0bc2b6c-ce87-44fb-9d69-25edfbe57f0f/volumes" Oct 30 20:30:12.979900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:12.979870 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp"] Oct 30 20:30:12.980265 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:30:12.979993 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef60147_b9a6_4e10_a979_ba5cbbcc4e92.slice/crio-69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102 WatchSource:0}: Error finding container 69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102: Status 404 returned error can't find the container with id 69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102 Oct 30 20:30:13.226313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.226225 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" containerID="cri-o://85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26" gracePeriod=45 Oct 30 20:30:13.231829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.231804 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" Oct 30 20:30:13.231968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.231827 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5" event={"ID":"32ef9ec4-a498-4de2-a9ac-368e26e5f67e","Type":"ContainerDied","Data":"053e212820b2c5df5e2fe65182784af985099e12afa56905d0667c16148806c3"} Oct 30 20:30:13.231968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.231905 5991 scope.go:122] "RemoveContainer" containerID="60e1ffa6ba8ecc349ab5fb30f5f09c69490b366a94daa9fe063aab7f7fdf8019" Oct 30 20:30:13.233439 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.233410 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" event={"ID":"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92","Type":"ContainerStarted","Data":"69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102"} Oct 30 20:30:13.346521 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.346493 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:30:13.350073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.350045 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7df9f446d6-vbrr5"] Oct 30 20:30:13.494862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.494831 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 20:30:13.562598 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562554 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\" (UniqueName: \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.562784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562638 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.562784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562716 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.562784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562748 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.562990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562811 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.562990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.562930 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\") pod \"ce6207f6-765e-4cb5-a8de-83106ab16872\" (UID: \"ce6207f6-765e-4cb5-a8de-83106ab16872\") " Oct 30 20:30:13.563247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.563209 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.563552 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.563526 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.563668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.563650 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.563920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.563892 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.565881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.565841 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:13.566640 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.566610 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.566640 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.566625 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk" pod "ce6207f6-765e-4cb5-a8de-83106ab16872" (UID: "ce6207f6-765e-4cb5-a8de-83106ab16872"). InnerVolumeSpecName "kube-api-access-jnjjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:13.665041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.665005 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.665041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.665038 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.665041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.665048 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6207f6-765e-4cb5-a8de-83106ab16872-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.665287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.665056 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjjk\" (UniqueName: \"kubernetes.io/projected/ce6207f6-765e-4cb5-a8de-83106ab16872-kube-api-access-jnjjk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.665287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:13.665065 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6207f6-765e-4cb5-a8de-83106ab16872-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:14.242789 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.242751 5991 generic.go:356] "Generic (PLEG): container finished" podID="7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" containerID="95ba085ca496c8e52c3c69f2fa7463a1e9925790a1a455a7c104c60cf8773493" exitCode=0 Oct 30 20:30:14.243244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.242837 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" event={"ID":"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92","Type":"ContainerDied","Data":"95ba085ca496c8e52c3c69f2fa7463a1e9925790a1a455a7c104c60cf8773493"} Oct 30 20:30:14.245452 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.245426 5991 generic.go:356] "Generic (PLEG): container finished" podID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerID="85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26" exitCode=0 Oct 30 20:30:14.245590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.245469 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" Oct 30 20:30:14.245590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.245505 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" event={"ID":"ce6207f6-765e-4cb5-a8de-83106ab16872","Type":"ContainerDied","Data":"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26"} Oct 30 20:30:14.245590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.245541 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" event={"ID":"ce6207f6-765e-4cb5-a8de-83106ab16872","Type":"ContainerDied","Data":"bf6a17b2e2506f45d3a9b5a05681ca1387eeaa9239a4856c969c39581d936104"} Oct 30 20:30:14.245590 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.245559 5991 scope.go:122] "RemoveContainer" containerID="85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26" Oct 30 20:30:14.255723 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.255699 5991 scope.go:122] "RemoveContainer" containerID="85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26" Oct 30 20:30:14.256088 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:14.256063 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26\": container with ID starting with 85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26 not found: ID does not exist" containerID="85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26" Oct 30 20:30:14.256178 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.256098 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26"} err="failed to get container status \"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26\": rpc error: code = NotFound desc = could not find container \"85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26\": container with ID starting with 85512d32c70364f666df95c3e8152c80a2a88872322a582cddb4eebe56aabf26 not found: ID does not exist" Oct 30 20:30:14.319919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.319870 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 20:30:14.324150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.324115 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz"] Oct 30 20:30:14.452899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.452852 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-2-w3-01f7d3-kserve-6fd77854fb-ff9fz" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:30:14.670977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.670892 5991 scope.go:122] "RemoveContainer" containerID="1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c" Oct 30 20:30:14.679341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.679309 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ef9ec4-a498-4de2-a9ac-368e26e5f67e" path="/var/lib/kubelet/pods/32ef9ec4-a498-4de2-a9ac-368e26e5f67e/volumes" Oct 30 20:30:14.679804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.679789 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" path="/var/lib/kubelet/pods/ce6207f6-765e-4cb5-a8de-83106ab16872/volumes" Oct 30 20:30:14.748055 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:14.748014 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:30:15.259690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.259593 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerStarted","Data":"40207e796a59182f934df5e35c7c71e738457c8baa8368dfbe4a8953135193ae"} Oct 30 20:30:15.311970 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.311916 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" podStartSLOduration=15.311902273 podStartE2EDuration="15.311902273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:15.309891474 +0000 UTC m=+3282.759384212" watchObservedRunningTime="2026-10-30 20:30:15.311902273 +0000 UTC m=+3282.761395040" Oct 30 20:30:15.442014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.441980 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:15.587529 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.587493 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\") pod \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " Oct 30 20:30:15.587746 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.587642 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\") pod \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " Oct 30 20:30:15.587746 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.587722 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\" (UniqueName: \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\") pod \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\" (UID: \"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92\") " Oct 30 20:30:15.587952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.587922 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca" pod "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" (UID: "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:15.588145 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.588128 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:15.589834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.589811 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca" pod "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" (UID: "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:15.589899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.589882 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88" pod "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" (UID: "7ef60147-b9a6-4e10-a979-ba5cbbcc4e92"). InnerVolumeSpecName "kube-api-access-2hr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:15.670908 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.670873 5991 scope.go:122] "RemoveContainer" containerID="969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5" Oct 30 20:30:15.689844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.689810 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:15.689844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.689839 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hr88\" (UniqueName: \"kubernetes.io/projected/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92-kube-api-access-2hr88\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:15.705475 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:30:15.705441 5991 logging.go:55] [core] [Channel #13289 SubChannel #13290]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.44:50052: connect: connection refused" Oct 30 20:30:15.889870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.889840 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 20:30:15.993212 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.993171 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\" (UniqueName: \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\") pod \"f006e551-4480-4475-a270-1d803ab8259e\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " Oct 30 20:30:15.993464 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.993387 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\") pod \"f006e551-4480-4475-a270-1d803ab8259e\" (UID: \"f006e551-4480-4475-a270-1d803ab8259e\") " Oct 30 20:30:15.993752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.993728 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume" pod "f006e551-4480-4475-a270-1d803ab8259e" (UID: "f006e551-4480-4475-a270-1d803ab8259e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:15.993828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.993816 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f006e551-4480-4475-a270-1d803ab8259e-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:15.995340 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:15.995314 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c" pod "f006e551-4480-4475-a270-1d803ab8259e" (UID: "f006e551-4480-4475-a270-1d803ab8259e"). InnerVolumeSpecName "kube-api-access-nm96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:16.094579 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.094511 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm96c\" (UniqueName: \"kubernetes.io/projected/f006e551-4480-4475-a270-1d803ab8259e-kube-api-access-nm96c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:16.268245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.268211 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" Oct 30 20:30:16.268735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.268214 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp" event={"ID":"7ef60147-b9a6-4e10-a979-ba5cbbcc4e92","Type":"ContainerDied","Data":"69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102"} Oct 30 20:30:16.268735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.268398 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69615caeca186cd039df03d88ebad6fcfc9ab3b4d21334010d560e75a1567102" Oct 30 20:30:16.270752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.270722 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerStarted","Data":"27ec725979a370d870d9aaecb47d95cdc8783cc2c481bc7a95916ebf909ef070"} Oct 30 20:30:16.272615 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.272585 5991 generic.go:356] "Generic (PLEG): container finished" podID="f006e551-4480-4475-a270-1d803ab8259e" containerID="12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae" exitCode=0 Oct 30 20:30:16.272751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.272636 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" Oct 30 20:30:16.272751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.272660 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" event={"ID":"f006e551-4480-4475-a270-1d803ab8259e","Type":"ContainerDied","Data":"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae"} Oct 30 20:30:16.272751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.272716 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" event={"ID":"f006e551-4480-4475-a270-1d803ab8259e","Type":"ContainerDied","Data":"4659796134e119cae8d1ad4e5242733b68c7d549a8c6fd1133fc9bea5baee767"} Oct 30 20:30:16.272751 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.272746 5991 scope.go:122] "RemoveContainer" containerID="12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae" Oct 30 20:30:16.278356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.278328 5991 generic.go:356] "Generic (PLEG): container finished" podID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerID="40207e796a59182f934df5e35c7c71e738457c8baa8368dfbe4a8953135193ae" exitCode=60 Oct 30 20:30:16.278558 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.278404 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerDied","Data":"40207e796a59182f934df5e35c7c71e738457c8baa8368dfbe4a8953135193ae"} Oct 30 20:30:16.278864 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.278845 5991 scope.go:122] "RemoveContainer" containerID="40207e796a59182f934df5e35c7c71e738457c8baa8368dfbe4a8953135193ae" Oct 30 20:30:16.279253 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:16.279186 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p_odh-ai-gateway-infra(02bea4eb-4c43-4dcb-bd45-48e9e0123a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" Oct 30 20:30:16.288013 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.287985 5991 scope.go:122] "RemoveContainer" containerID="12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae" Oct 30 20:30:16.288437 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:16.288404 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae\": container with ID starting with 12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae not found: ID does not exist" containerID="12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae" Oct 30 20:30:16.288545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.288452 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae"} err="failed to get container status \"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae\": rpc error: code = NotFound desc = could not find container \"12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae\": container with ID starting with 12c52f75c5b91a213c9c2e2a085b1e31608fd4228e43a576875c0814b74f27ae not found: ID does not exist" Oct 30 20:30:16.288545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.288473 5991 scope.go:122] "RemoveContainer" containerID="1f03f54d8d8b72408eb35b4da88449b1b71269e9507823705ad2b746bc8fa64c" Oct 30 20:30:16.313640 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.313574 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" podStartSLOduration=16.313558644 podStartE2EDuration="16.313558644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:16.311732913 +0000 UTC m=+3283.761225638" watchObservedRunningTime="2026-10-30 20:30:16.313558644 +0000 UTC m=+3283.763051369" Oct 30 20:30:16.320729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.320694 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:16.415798 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.415767 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 20:30:16.422613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.422582 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8"] Oct 30 20:30:16.679110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.679077 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f006e551-4480-4475-a270-1d803ab8259e" path="/var/lib/kubelet/pods/f006e551-4480-4475-a270-1d803ab8259e/volumes" Oct 30 20:30:16.705356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:16.705309 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-284bf6-65ffbd886-2d7j8" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded" Oct 30 20:30:17.286691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.286658 5991 generic.go:356] "Generic (PLEG): container finished" podID="b156438c-2e22-43d0-accb-e534fa7843ed" containerID="27ec725979a370d870d9aaecb47d95cdc8783cc2c481bc7a95916ebf909ef070" exitCode=60 Oct 30 20:30:17.287170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.286727 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerDied","Data":"27ec725979a370d870d9aaecb47d95cdc8783cc2c481bc7a95916ebf909ef070"} Oct 30 20:30:17.287170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.286791 5991 scope.go:122] "RemoveContainer" containerID="969499f8358bf77fbd6299963e9ea569c564f68b1a97d87dceee174ad0296bc5" Oct 30 20:30:17.287277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.287224 5991 scope.go:122] "RemoveContainer" containerID="27ec725979a370d870d9aaecb47d95cdc8783cc2c481bc7a95916ebf909ef070" Oct 30 20:30:17.287598 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:17.287571 5991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm_odh-ai-gateway-infra(b156438c-2e22-43d0-accb-e534fa7843ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" Oct 30 20:30:17.326515 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.326475 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:17.455458 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.455425 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:17.509494 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.509458 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\" (UniqueName: \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\") pod \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " Oct 30 20:30:17.509696 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.509600 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\") pod \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\" (UID: \"02bea4eb-4c43-4dcb-bd45-48e9e0123a28\") " Oct 30 20:30:17.510027 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.509998 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca" pod "02bea4eb-4c43-4dcb-bd45-48e9e0123a28" (UID: "02bea4eb-4c43-4dcb-bd45-48e9e0123a28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:17.512245 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.512219 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5" pod "02bea4eb-4c43-4dcb-bd45-48e9e0123a28" (UID: "02bea4eb-4c43-4dcb-bd45-48e9e0123a28"). InnerVolumeSpecName "kube-api-access-vrzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:17.610895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.610798 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrzs5\" (UniqueName: \"kubernetes.io/projected/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-kube-api-access-vrzs5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:17.610895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.610838 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bea4eb-4c43-4dcb-bd45-48e9e0123a28-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:17.722814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:17.722773 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:18.300247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.300209 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" Oct 30 20:30:18.300247 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.300233 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p" event={"ID":"02bea4eb-4c43-4dcb-bd45-48e9e0123a28","Type":"ContainerDied","Data":"4405335e2dd22ad7b84af150572574b6ce1ade23d4d92770f68d0279fb365e11"} Oct 30 20:30:18.300889 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.300337 5991 scope.go:122] "RemoveContainer" containerID="40207e796a59182f934df5e35c7c71e738457c8baa8368dfbe4a8953135193ae" Oct 30 20:30:18.399426 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.399327 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:18.408043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.408001 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889870-rbc7p"] Oct 30 20:30:18.482261 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.482222 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:18.624940 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.624840 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\") pod \"b156438c-2e22-43d0-accb-e534fa7843ed\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " Oct 30 20:30:18.625111 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.625047 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\" (UniqueName: \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\") pod \"b156438c-2e22-43d0-accb-e534fa7843ed\" (UID: \"b156438c-2e22-43d0-accb-e534fa7843ed\") " Oct 30 20:30:18.625315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.625288 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca" pod "b156438c-2e22-43d0-accb-e534fa7843ed" (UID: "b156438c-2e22-43d0-accb-e534fa7843ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:18.627251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.627218 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9" pod "b156438c-2e22-43d0-accb-e534fa7843ed" (UID: "b156438c-2e22-43d0-accb-e534fa7843ed"). InnerVolumeSpecName "kube-api-access-ftvt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:18.678847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.678807 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" path="/var/lib/kubelet/pods/02bea4eb-4c43-4dcb-bd45-48e9e0123a28/volumes" Oct 30 20:30:18.726947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.726897 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvt9\" (UniqueName: \"kubernetes.io/projected/b156438c-2e22-43d0-accb-e534fa7843ed-kube-api-access-ftvt9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:18.726947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:18.726936 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b156438c-2e22-43d0-accb-e534fa7843ed-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:19.312070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:19.312022 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" Oct 30 20:30:19.312070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:19.312057 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm" event={"ID":"b156438c-2e22-43d0-accb-e534fa7843ed","Type":"ContainerDied","Data":"f69fdf8d716058355b9aacf567d14ebd5273c605ddb621d7086eddcc06c0519d"} Oct 30 20:30:19.312747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:19.312132 5991 scope.go:122] "RemoveContainer" containerID="27ec725979a370d870d9aaecb47d95cdc8783cc2c481bc7a95916ebf909ef070" Oct 30 20:30:19.379414 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:19.379348 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:19.388957 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:19.388912 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5c875d-29889870-z4cbm"] Oct 30 20:30:20.678401 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:20.678331 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" path="/var/lib/kubelet/pods/b156438c-2e22-43d0-accb-e534fa7843ed/volumes" Oct 30 20:30:20.871009 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:20.870973 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:30:27.723036 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:27.722999 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:28.435112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:28.435076 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:30:28.920569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:28.920539 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 20:30:29.049097 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.049008 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\") pod \"03cc3275-277b-4536-8ae6-7db2e9126f37\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " Oct 30 20:30:29.049284 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.049194 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\" (UniqueName: \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\") pod \"03cc3275-277b-4536-8ae6-7db2e9126f37\" (UID: \"03cc3275-277b-4536-8ae6-7db2e9126f37\") " Oct 30 20:30:29.049423 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.049397 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume" pod "03cc3275-277b-4536-8ae6-7db2e9126f37" (UID: "03cc3275-277b-4536-8ae6-7db2e9126f37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:29.049649 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.049629 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc3275-277b-4536-8ae6-7db2e9126f37-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:29.051257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.051235 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks" pod "03cc3275-277b-4536-8ae6-7db2e9126f37" (UID: "03cc3275-277b-4536-8ae6-7db2e9126f37"). InnerVolumeSpecName "kube-api-access-949ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:29.151032 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.150982 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949ks\" (UniqueName: \"kubernetes.io/projected/03cc3275-277b-4536-8ae6-7db2e9126f37-kube-api-access-949ks\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:29.384819 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.384727 5991 generic.go:356] "Generic (PLEG): container finished" podID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerID="a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e" exitCode=0 Oct 30 20:30:29.385028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.384800 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" Oct 30 20:30:29.385153 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.384829 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" event={"ID":"03cc3275-277b-4536-8ae6-7db2e9126f37","Type":"ContainerDied","Data":"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e"} Oct 30 20:30:29.385216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.385165 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9" event={"ID":"03cc3275-277b-4536-8ae6-7db2e9126f37","Type":"ContainerDied","Data":"34a8e092d5de4637ccd934315c2ab838d1ffaee02c7e651b4b9a47a03ed3d13f"} Oct 30 20:30:29.385216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.385189 5991 scope.go:122] "RemoveContainer" containerID="a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e" Oct 30 20:30:29.395756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.395734 5991 scope.go:122] "RemoveContainer" containerID="a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e" Oct 30 20:30:29.396071 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:29.396049 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e\": container with ID starting with a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e not found: ID does not exist" containerID="a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e" Oct 30 20:30:29.396120 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.396078 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e"} err="failed to get container status \"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e\": rpc error: code = NotFound desc = could not find container \"a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e\": container with ID starting with a1597d38292be779fc148874be797baf8ed9188abe42157fb7fa6f61ffefc56e not found: ID does not exist" Oct 30 20:30:29.443343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.443305 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 20:30:29.453482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.453444 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-01f7d3-54bd49ddfd-scmq9"] Oct 30 20:30:29.594949 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.594911 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:30:29.643182 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:29.643086 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:30:30.679777 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:30.679738 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" path="/var/lib/kubelet/pods/03cc3275-277b-4536-8ae6-7db2e9126f37/volumes" Oct 30 20:30:31.154527 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.154493 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 20:30:31.154744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.154706 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" podUID="b97b046d-1071-4817-8c94-403ecb2ed589" containerName="maas-api" containerID="cri-o://5a939671ce154e141a0f29e58e4129d581b58a3dd332d8144885504b61064c52" gracePeriod=30 Oct 30 20:30:31.401695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.401657 5991 generic.go:356] "Generic (PLEG): container finished" podID="b97b046d-1071-4817-8c94-403ecb2ed589" containerID="5a939671ce154e141a0f29e58e4129d581b58a3dd332d8144885504b61064c52" exitCode=0 Oct 30 20:30:31.401856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.401727 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" event={"ID":"b97b046d-1071-4817-8c94-403ecb2ed589","Type":"ContainerDied","Data":"5a939671ce154e141a0f29e58e4129d581b58a3dd332d8144885504b61064c52"} Oct 30 20:30:31.401856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.401790 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" event={"ID":"b97b046d-1071-4817-8c94-403ecb2ed589","Type":"ContainerDied","Data":"ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71"} Oct 30 20:30:31.401856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.401810 5991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae73280729abc6f04c91afe253d1fb01d2fbc3ae7d46576cd5229183337f4a71" Oct 30 20:30:31.421916 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.421849 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 20:30:31.486115 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.486077 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\") pod \"b97b046d-1071-4817-8c94-403ecb2ed589\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " Oct 30 20:30:31.486347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.486174 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\" (UniqueName: \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\") pod \"b97b046d-1071-4817-8c94-403ecb2ed589\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " Oct 30 20:30:31.486347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.486261 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\") pod \"b97b046d-1071-4817-8c94-403ecb2ed589\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " Oct 30 20:30:31.486347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.486297 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\") pod \"b97b046d-1071-4817-8c94-403ecb2ed589\" (UID: \"b97b046d-1071-4817-8c94-403ecb2ed589\") " Oct 30 20:30:31.487064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.487014 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca" pod "b97b046d-1071-4817-8c94-403ecb2ed589" (UID: "b97b046d-1071-4817-8c94-403ecb2ed589"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:31.488929 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.488895 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls" pod "b97b046d-1071-4817-8c94-403ecb2ed589" (UID: "b97b046d-1071-4817-8c94-403ecb2ed589"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:31.489067 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.488990 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs" pod "b97b046d-1071-4817-8c94-403ecb2ed589" (UID: "b97b046d-1071-4817-8c94-403ecb2ed589"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:31.492899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.492858 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd" pod "b97b046d-1071-4817-8c94-403ecb2ed589" (UID: "b97b046d-1071-4817-8c94-403ecb2ed589"). InnerVolumeSpecName "kube-api-access-t68bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:31.588587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.588530 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:31.588587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.588583 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68bd\" (UniqueName: \"kubernetes.io/projected/b97b046d-1071-4817-8c94-403ecb2ed589-kube-api-access-t68bd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:31.588587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.588595 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97b046d-1071-4817-8c94-403ecb2ed589-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:31.588587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.588604 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97b046d-1071-4817-8c94-403ecb2ed589-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:31.658405 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.658344 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:30:31.658658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:31.658630 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" podUID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" containerName="maas-api" containerID="cri-o://58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af" gracePeriod=30 Oct 30 20:30:32.113914 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.113882 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:30:32.196087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.195989 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\") pod \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " Oct 30 20:30:32.196087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.196060 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\") pod \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " Oct 30 20:30:32.196347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.196106 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\") pod \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " Oct 30 20:30:32.196347 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.196170 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\" (UniqueName: \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\") pod \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\" (UID: \"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2\") " Oct 30 20:30:32.197115 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.197065 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca" pod "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" (UID: "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:32.198645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.198598 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls" pod "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" (UID: "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:32.198955 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.198924 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55" pod "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" (UID: "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2"). InnerVolumeSpecName "kube-api-access-qsz55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:32.199509 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.199466 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs" pod "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" (UID: "f33cfc0d-ddbf-40df-bb30-e636a9a3aee2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:32.298037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.297995 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.298037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.298028 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.298037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.298038 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.298037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.298048 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsz55\" (UniqueName: \"kubernetes.io/projected/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2-kube-api-access-qsz55\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.409480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409438 5991 generic.go:356] "Generic (PLEG): container finished" podID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" containerID="58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af" exitCode=0 Oct 30 20:30:32.409668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409488 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" Oct 30 20:30:32.409668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409526 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" event={"ID":"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2","Type":"ContainerDied","Data":"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af"} Oct 30 20:30:32.409668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409588 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr" event={"ID":"f33cfc0d-ddbf-40df-bb30-e636a9a3aee2","Type":"ContainerDied","Data":"c709f7c03e6d618a9642b0d6965c156855ea0d7164dc9c0ce3cfec8fd826c10b"} Oct 30 20:30:32.409668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409616 5991 scope.go:122] "RemoveContainer" containerID="58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af" Oct 30 20:30:32.409914 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.409727 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp" Oct 30 20:30:32.421471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.421447 5991 scope.go:122] "RemoveContainer" containerID="58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af" Oct 30 20:30:32.421831 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:32.421804 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af\": container with ID starting with 58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af not found: ID does not exist" containerID="58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af" Oct 30 20:30:32.421920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.421843 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af"} err="failed to get container status \"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af\": rpc error: code = NotFound desc = could not find container \"58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af\": container with ID starting with 58322a869bdb3d62ac22fb7cdb1516c6ead4257ca3b05e8e6c884ff7c02449af not found: ID does not exist" Oct 30 20:30:32.502706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.502666 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 20:30:32.516014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.515963 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-01f7d3-7b9bf5d859-cqjtp"] Oct 30 20:30:32.556501 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.556463 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:30:32.599310 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.599276 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 20:30:32.601244 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.601219 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77ac5a-86c9d94b89-m87qr"] Oct 30 20:30:32.683041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.682997 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97b046d-1071-4817-8c94-403ecb2ed589" path="/var/lib/kubelet/pods/b97b046d-1071-4817-8c94-403ecb2ed589/volumes" Oct 30 20:30:32.683784 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:32.683750 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" path="/var/lib/kubelet/pods/f33cfc0d-ddbf-40df-bb30-e636a9a3aee2/volumes" Oct 30 20:30:33.294083 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:33.294050 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:30:33.327319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:33.327277 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:30:33.435779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:33.435712 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" containerID="cri-o://acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422" gracePeriod=25 Oct 30 20:30:34.596077 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:34.596023 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" containerID="cri-o://ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86" gracePeriod=25 Oct 30 20:30:34.644035 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:34.643976 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" containerID="cri-o://244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb" gracePeriod=25 Oct 30 20:30:34.723796 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:34.723762 5991 scope.go:122] "RemoveContainer" containerID="221bd53af5ba42dacdbbf24ff25feeb531c836b27988d7bc95607fa86c433a42" Oct 30 20:30:34.735202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:34.735169 5991 scope.go:122] "RemoveContainer" containerID="5a939671ce154e141a0f29e58e4129d581b58a3dd332d8144885504b61064c52" Oct 30 20:30:36.248194 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:36.248145 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:37.131899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:37.131856 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:37.131899 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:37.131893 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:37.600297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:37.600250 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" containerID="cri-o://3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a" gracePeriod=25 Oct 30 20:30:38.523798 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:38.523759 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:45.217457 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:45.217421 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 20:30:45.217844 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:45.217626 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" containerID="cri-o://ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997" gracePeriod=30 Oct 30 20:30:46.247747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:46.247709 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:47.131003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:47.130966 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:47.131196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:47.130977 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:48.524560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:48.524522 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:49.436587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.436541 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 20:30:49.436894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.436860 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" podUID="3b25294d-fe92-4a3e-becc-db8ef0331beb" containerName="maas-api" containerID="cri-o://4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859" gracePeriod=30 Oct 30 20:30:49.713695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.713663 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 20:30:49.805010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.804973 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\") pod \"3b25294d-fe92-4a3e-becc-db8ef0331beb\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " Oct 30 20:30:49.805223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.805054 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\") pod \"3b25294d-fe92-4a3e-becc-db8ef0331beb\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " Oct 30 20:30:49.805223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.805126 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") pod \"3b25294d-fe92-4a3e-becc-db8ef0331beb\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " Oct 30 20:30:49.805223 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.805211 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\" (UniqueName: \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\") pod \"3b25294d-fe92-4a3e-becc-db8ef0331beb\" (UID: \"3b25294d-fe92-4a3e-becc-db8ef0331beb\") " Oct 30 20:30:49.805553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.805521 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca" pod "3b25294d-fe92-4a3e-becc-db8ef0331beb" (UID: "3b25294d-fe92-4a3e-becc-db8ef0331beb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:49.806042 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.805960 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25294d-fe92-4a3e-becc-db8ef0331beb-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:49.807611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.807578 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs" pod "3b25294d-fe92-4a3e-becc-db8ef0331beb" (UID: "3b25294d-fe92-4a3e-becc-db8ef0331beb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:49.807885 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.807864 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls" pod "3b25294d-fe92-4a3e-becc-db8ef0331beb" (UID: "3b25294d-fe92-4a3e-becc-db8ef0331beb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:49.808064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.808048 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6" pod "3b25294d-fe92-4a3e-becc-db8ef0331beb" (UID: "3b25294d-fe92-4a3e-becc-db8ef0331beb"). InnerVolumeSpecName "kube-api-access-7rsr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:49.907539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.907502 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rsr6\" (UniqueName: \"kubernetes.io/projected/3b25294d-fe92-4a3e-becc-db8ef0331beb-kube-api-access-7rsr6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:49.907539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.907536 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:49.907539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:49.907545 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b25294d-fe92-4a3e-becc-db8ef0331beb-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.342112 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.342061 5991 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:30:50.342313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.342124 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:30:50.497865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.497789 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 20:30:50.524587 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.524549 5991 generic.go:356] "Generic (PLEG): container finished" podID="3b25294d-fe92-4a3e-becc-db8ef0331beb" containerID="4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859" exitCode=0 Oct 30 20:30:50.524779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.524594 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" Oct 30 20:30:50.524779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.524605 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" event={"ID":"3b25294d-fe92-4a3e-becc-db8ef0331beb","Type":"ContainerDied","Data":"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859"} Oct 30 20:30:50.524779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.524665 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg" event={"ID":"3b25294d-fe92-4a3e-becc-db8ef0331beb","Type":"ContainerDied","Data":"c85a159d4bdf3458cc1e9d8f874e098b059393bd105fc921278c1589476f2a0c"} Oct 30 20:30:50.524779 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.524700 5991 scope.go:122] "RemoveContainer" containerID="4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859" Oct 30 20:30:50.526140 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.526116 5991 generic.go:356] "Generic (PLEG): container finished" podID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerID="ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997" exitCode=0 Oct 30 20:30:50.526241 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.526173 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" Oct 30 20:30:50.526241 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.526191 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" event={"ID":"ee65e032-5d15-4462-ac7f-300575e6fc4d","Type":"ContainerDied","Data":"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997"} Oct 30 20:30:50.526406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.526254 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k" event={"ID":"ee65e032-5d15-4462-ac7f-300575e6fc4d","Type":"ContainerDied","Data":"d729c033bb8327dd611bae0aee1cf30baf54cc8d64c60d7441be0673322d8869"} Oct 30 20:30:50.534729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.534708 5991 scope.go:122] "RemoveContainer" containerID="4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859" Oct 30 20:30:50.535074 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:50.535053 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859\": container with ID starting with 4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859 not found: ID does not exist" containerID="4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859" Oct 30 20:30:50.535121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.535085 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859"} err="failed to get container status \"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859\": rpc error: code = NotFound desc = could not find container \"4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859\": container with ID starting with 4b2f2821a0a7e1c22bfebfe495b35ca7a534e3ae4204c16e28eb55739ec14859 not found: ID does not exist" Oct 30 20:30:50.535121 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.535104 5991 scope.go:122] "RemoveContainer" containerID="ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997" Oct 30 20:30:50.544552 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.544526 5991 scope.go:122] "RemoveContainer" containerID="ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997" Oct 30 20:30:50.544881 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:50.544864 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997\": container with ID starting with ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997 not found: ID does not exist" containerID="ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997" Oct 30 20:30:50.544947 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.544894 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997"} err="failed to get container status \"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997\": rpc error: code = NotFound desc = could not find container \"ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997\": container with ID starting with ad165bf66f9440913b74181292e35fc9baa8aee600a0b13793b678f0dd784997 not found: ID does not exist" Oct 30 20:30:50.617351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617310 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617406 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617438 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617502 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617540 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617564 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617628 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617600 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617651 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617737 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617741 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617826 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617891 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.617981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.617902 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\") pod \"ee65e032-5d15-4462-ac7f-300575e6fc4d\" (UID: \"ee65e032-5d15-4462-ac7f-300575e6fc4d\") " Oct 30 20:30:50.618268 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618138 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:50.618549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618525 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee65e032-5d15-4462-ac7f-300575e6fc4d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.618829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618554 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.618829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618568 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.618829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618583 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.618829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.618597 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.620076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.620050 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:50.620166 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.620121 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "kube-api-access-d74qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:50.620242 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.620228 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:30:50.620314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.620302 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy" pod "ee65e032-5d15-4462-ac7f-300575e6fc4d" (UID: "ee65e032-5d15-4462-ac7f-300575e6fc4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.642743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.642708 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 20:30:50.652549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.652503 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5c2e32-749ddbdf54-ntfxg"] Oct 30 20:30:50.679640 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.679589 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b25294d-fe92-4a3e-becc-db8ef0331beb" path="/var/lib/kubelet/pods/3b25294d-fe92-4a3e-becc-db8ef0331beb/volumes" Oct 30 20:30:50.720106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.720061 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.720106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.720092 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74qk\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-kube-api-access-d74qk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.720106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.720107 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.720630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.720120 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee65e032-5d15-4462-ac7f-300575e6fc4d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.881854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.871268 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 20:30:50.881854 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:50.875062 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-8xr2k"] Oct 30 20:30:50.882782 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:50.882743 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:30:52.679442 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:52.679354 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" path="/var/lib/kubelet/pods/ee65e032-5d15-4462-ac7f-300575e6fc4d/volumes" Oct 30 20:30:53.710932 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.710862 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:30:53.857968 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.857931 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\" (UniqueName: \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\") pod \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " Oct 30 20:30:53.858163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.858097 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\") pod \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\" (UID: \"13c78352-08a5-4222-a2e0-5849f9ae8fcd\") " Oct 30 20:30:53.858607 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.858566 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume" pod "13c78352-08a5-4222-a2e0-5849f9ae8fcd" (UID: "13c78352-08a5-4222-a2e0-5849f9ae8fcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:53.860096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.860072 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z" pod "13c78352-08a5-4222-a2e0-5849f9ae8fcd" (UID: "13c78352-08a5-4222-a2e0-5849f9ae8fcd"). InnerVolumeSpecName "kube-api-access-snv9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:53.960479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.960439 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snv9z\" (UniqueName: \"kubernetes.io/projected/13c78352-08a5-4222-a2e0-5849f9ae8fcd-kube-api-access-snv9z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:53.960479 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:53.960475 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c78352-08a5-4222-a2e0-5849f9ae8fcd-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:54.554198 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.554165 5991 generic.go:356] "Generic (PLEG): container finished" podID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerID="acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422" exitCode=0 Oct 30 20:30:54.554454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.554219 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" Oct 30 20:30:54.554454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.554248 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" event={"ID":"13c78352-08a5-4222-a2e0-5849f9ae8fcd","Type":"ContainerDied","Data":"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422"} Oct 30 20:30:54.554454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.554315 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk" event={"ID":"13c78352-08a5-4222-a2e0-5849f9ae8fcd","Type":"ContainerDied","Data":"595dcd97cb16ffda610a2b14850864c7350d2aa3e92c3134d0dd3d4e91633525"} Oct 30 20:30:54.554454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.554348 5991 scope.go:122] "RemoveContainer" containerID="acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422" Oct 30 20:30:54.564294 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.564134 5991 scope.go:122] "RemoveContainer" containerID="acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422" Oct 30 20:30:54.564429 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:54.564412 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422\": container with ID starting with acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422 not found: ID does not exist" containerID="acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422" Oct 30 20:30:54.564484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.564437 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422"} err="failed to get container status \"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422\": rpc error: code = NotFound desc = could not find container \"acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422\": container with ID starting with acaeee89bf6dc04157d378898419c7edb1e746417bf37cf469cfb0dd69d76422 not found: ID does not exist" Oct 30 20:30:54.605629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.605600 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:30:54.608754 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.608726 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5314c2-cf849dd7-5mpbk"] Oct 30 20:30:54.678393 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.678344 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" path="/var/lib/kubelet/pods/13c78352-08a5-4222-a2e0-5849f9ae8fcd/volumes" Oct 30 20:30:54.903586 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.903558 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:30:54.932989 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:54.932964 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:30:55.074059 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.073968 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\" (UniqueName: \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\") pod \"901efb89-7a25-40ea-9fb5-8d3f95286f72\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " Oct 30 20:30:55.074059 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.074013 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\") pod \"f02f4a3f-4d82-4688-8b80-83b7d2332378\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " Oct 30 20:30:55.074059 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.074037 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\" (UniqueName: \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\") pod \"f02f4a3f-4d82-4688-8b80-83b7d2332378\" (UID: \"f02f4a3f-4d82-4688-8b80-83b7d2332378\") " Oct 30 20:30:55.074304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.074109 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\") pod \"901efb89-7a25-40ea-9fb5-8d3f95286f72\" (UID: \"901efb89-7a25-40ea-9fb5-8d3f95286f72\") " Oct 30 20:30:55.074571 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.074539 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume" pod "f02f4a3f-4d82-4688-8b80-83b7d2332378" (UID: "f02f4a3f-4d82-4688-8b80-83b7d2332378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:55.074706 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.074591 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume" pod "901efb89-7a25-40ea-9fb5-8d3f95286f72" (UID: "901efb89-7a25-40ea-9fb5-8d3f95286f72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:55.076257 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.076232 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf" pod "f02f4a3f-4d82-4688-8b80-83b7d2332378" (UID: "f02f4a3f-4d82-4688-8b80-83b7d2332378"). InnerVolumeSpecName "kube-api-access-qcrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:55.076391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.076327 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz" pod "901efb89-7a25-40ea-9fb5-8d3f95286f72" (UID: "901efb89-7a25-40ea-9fb5-8d3f95286f72"). InnerVolumeSpecName "kube-api-access-8h6bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:55.176221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.176185 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6bz\" (UniqueName: \"kubernetes.io/projected/901efb89-7a25-40ea-9fb5-8d3f95286f72-kube-api-access-8h6bz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:55.176221 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.176217 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f4a3f-4d82-4688-8b80-83b7d2332378-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:55.176453 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.176234 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrhf\" (UniqueName: \"kubernetes.io/projected/f02f4a3f-4d82-4688-8b80-83b7d2332378-kube-api-access-qcrhf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:55.176453 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.176243 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901efb89-7a25-40ea-9fb5-8d3f95286f72-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:55.567354 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.567313 5991 generic.go:356] "Generic (PLEG): container finished" podID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerID="244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb" exitCode=0 Oct 30 20:30:55.567583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.567394 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" Oct 30 20:30:55.567583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.567403 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" event={"ID":"f02f4a3f-4d82-4688-8b80-83b7d2332378","Type":"ContainerDied","Data":"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb"} Oct 30 20:30:55.567583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.567477 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22" event={"ID":"f02f4a3f-4d82-4688-8b80-83b7d2332378","Type":"ContainerDied","Data":"8b18fa9d55d17b59d2ce317b7b68e854814d995643ef9b2cba510e4ccc99138b"} Oct 30 20:30:55.567583 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.567503 5991 scope.go:122] "RemoveContainer" containerID="244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb" Oct 30 20:30:55.568877 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.568851 5991 generic.go:356] "Generic (PLEG): container finished" podID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerID="ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86" exitCode=0 Oct 30 20:30:55.569002 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.568884 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" event={"ID":"901efb89-7a25-40ea-9fb5-8d3f95286f72","Type":"ContainerDied","Data":"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86"} Oct 30 20:30:55.569002 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.568925 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" Oct 30 20:30:55.569002 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.568940 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7" event={"ID":"901efb89-7a25-40ea-9fb5-8d3f95286f72","Type":"ContainerDied","Data":"193964152ac7d1e8c10a9c65af3e20ae3c3181f6688ebb9614dc3bb864d618aa"} Oct 30 20:30:55.579870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.579843 5991 scope.go:122] "RemoveContainer" containerID="244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb" Oct 30 20:30:55.580342 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:55.580315 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb\": container with ID starting with 244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb not found: ID does not exist" containerID="244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb" Oct 30 20:30:55.580522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.580494 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb"} err="failed to get container status \"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb\": rpc error: code = NotFound desc = could not find container \"244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb\": container with ID starting with 244b7a7bc6688cc36f97cea627daf257338a9844020264fec78d64a87c4385cb not found: ID does not exist" Oct 30 20:30:55.580621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.580605 5991 scope.go:122] "RemoveContainer" containerID="ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86" Oct 30 20:30:55.591630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.591603 5991 scope.go:122] "RemoveContainer" containerID="ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86" Oct 30 20:30:55.591986 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:55.591958 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86\": container with ID starting with ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86 not found: ID does not exist" containerID="ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86" Oct 30 20:30:55.592095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.591993 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86"} err="failed to get container status \"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86\": rpc error: code = NotFound desc = could not find container \"ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86\": container with ID starting with ff63a881147bc6831bea1562b3adc77a09e87c01df75737d933d2c33ad5a4d86 not found: ID does not exist" Oct 30 20:30:55.623430 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.623396 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:30:55.628079 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.628048 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77ac5a-54859f47f4-ccrp7"] Oct 30 20:30:55.672208 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.672175 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:30:55.677421 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:55.677355 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77ac5a-7d6b494d99-ctv22"] Oct 30 20:30:56.677637 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:56.677602 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" path="/var/lib/kubelet/pods/901efb89-7a25-40ea-9fb5-8d3f95286f72/volumes" Oct 30 20:30:56.678089 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:56.678070 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" path="/var/lib/kubelet/pods/f02f4a3f-4d82-4688-8b80-83b7d2332378/volumes" Oct 30 20:30:57.875251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:57.875178 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 20:30:57.907101 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:57.907062 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\" (UniqueName: \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\") pod \"d3c6216b-e0b5-4858-8bc7-72793593fa13\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " Oct 30 20:30:57.907259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:57.907198 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\") pod \"d3c6216b-e0b5-4858-8bc7-72793593fa13\" (UID: \"d3c6216b-e0b5-4858-8bc7-72793593fa13\") " Oct 30 20:30:57.907621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:57.907595 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume" pod "d3c6216b-e0b5-4858-8bc7-72793593fa13" (UID: "d3c6216b-e0b5-4858-8bc7-72793593fa13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:57.909478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:57.909448 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw" pod "d3c6216b-e0b5-4858-8bc7-72793593fa13" (UID: "d3c6216b-e0b5-4858-8bc7-72793593fa13"). InnerVolumeSpecName "kube-api-access-tc5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:58.008935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.008891 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5lw\" (UniqueName: \"kubernetes.io/projected/d3c6216b-e0b5-4858-8bc7-72793593fa13-kube-api-access-tc5lw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:58.008935 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.008928 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c6216b-e0b5-4858-8bc7-72793593fa13-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:30:58.591500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.591459 5991 generic.go:356] "Generic (PLEG): container finished" podID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerID="3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a" exitCode=0 Oct 30 20:30:58.591688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.591508 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" Oct 30 20:30:58.591688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.591520 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" event={"ID":"d3c6216b-e0b5-4858-8bc7-72793593fa13","Type":"ContainerDied","Data":"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a"} Oct 30 20:30:58.591688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.591574 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr" event={"ID":"d3c6216b-e0b5-4858-8bc7-72793593fa13","Type":"ContainerDied","Data":"bad885ee752aa646c696339e147c3a6a0f29ac937db2d0e6b67a18aeee1160b8"} Oct 30 20:30:58.591688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.591605 5991 scope.go:122] "RemoveContainer" containerID="3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a" Oct 30 20:30:58.602009 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.601989 5991 scope.go:122] "RemoveContainer" containerID="3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a" Oct 30 20:30:58.602323 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:30:58.602301 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a\": container with ID starting with 3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a not found: ID does not exist" containerID="3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a" Oct 30 20:30:58.602398 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.602337 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a"} err="failed to get container status \"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a\": rpc error: code = NotFound desc = could not find container \"3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a\": container with ID starting with 3a0ed81e46afe44eb7906798f6a982c3ef97362eb9fa6ba37ff4c52dd9e1231a not found: ID does not exist" Oct 30 20:30:58.647318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.647284 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 20:30:58.659438 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.659395 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5c2e32-7b6b549d8d-45dsr"] Oct 30 20:30:58.677471 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.677439 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" path="/var/lib/kubelet/pods/d3c6216b-e0b5-4858-8bc7-72793593fa13/volumes" Oct 30 20:30:58.862206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.862123 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:30:58.895553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:30:58.895512 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:31:00.651574 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:00.651537 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:31:03.863303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:03.863255 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" containerID="cri-o://5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff" gracePeriod=25 Oct 30 20:31:03.896502 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:03.896441 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" containerID="cri-o://b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915" gracePeriod=25 Oct 30 20:31:04.426219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.426182 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:31:04.426486 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.426428 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" podUID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" containerName="maas-api" containerID="cri-o://7e6c9b768f7ba6db5e1dc967c88370f4a47c1f1756decc8f5d8d496f4cb074f4" gracePeriod=30 Oct 30 20:31:04.630676 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.630632 5991 generic.go:356] "Generic (PLEG): container finished" podID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" containerID="7e6c9b768f7ba6db5e1dc967c88370f4a47c1f1756decc8f5d8d496f4cb074f4" exitCode=0 Oct 30 20:31:04.630878 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.630705 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" event={"ID":"963d4ab2-b668-41bc-9ba8-7114f8f38a15","Type":"ContainerDied","Data":"7e6c9b768f7ba6db5e1dc967c88370f4a47c1f1756decc8f5d8d496f4cb074f4"} Oct 30 20:31:04.692690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.692616 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:31:04.779850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.779814 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\") pod \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " Oct 30 20:31:04.780060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.779907 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\") pod \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " Oct 30 20:31:04.780060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.779962 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\" (UniqueName: \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\") pod \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " Oct 30 20:31:04.780060 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.780008 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\") pod \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\" (UID: \"963d4ab2-b668-41bc-9ba8-7114f8f38a15\") " Oct 30 20:31:04.780339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.780308 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca" pod "963d4ab2-b668-41bc-9ba8-7114f8f38a15" (UID: "963d4ab2-b668-41bc-9ba8-7114f8f38a15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:04.780518 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.780501 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d4ab2-b668-41bc-9ba8-7114f8f38a15-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:04.782127 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.782102 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs" pod "963d4ab2-b668-41bc-9ba8-7114f8f38a15" (UID: "963d4ab2-b668-41bc-9ba8-7114f8f38a15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:04.782233 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.782144 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h" pod "963d4ab2-b668-41bc-9ba8-7114f8f38a15" (UID: "963d4ab2-b668-41bc-9ba8-7114f8f38a15"). InnerVolumeSpecName "kube-api-access-lm72h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:04.782289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.782241 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls" pod "963d4ab2-b668-41bc-9ba8-7114f8f38a15" (UID: "963d4ab2-b668-41bc-9ba8-7114f8f38a15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:04.881629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.881592 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm72h\" (UniqueName: \"kubernetes.io/projected/963d4ab2-b668-41bc-9ba8-7114f8f38a15-kube-api-access-lm72h\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:04.881629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.881627 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:04.882087 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:04.881641 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963d4ab2-b668-41bc-9ba8-7114f8f38a15-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:05.638990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:05.638955 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" Oct 30 20:31:05.638990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:05.638981 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n" event={"ID":"963d4ab2-b668-41bc-9ba8-7114f8f38a15","Type":"ContainerDied","Data":"90c48aaf0140b05f004de58bd985b2c22337639ae41908aa16e85f4e68c415b8"} Oct 30 20:31:05.639282 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:05.639041 5991 scope.go:122] "RemoveContainer" containerID="7e6c9b768f7ba6db5e1dc967c88370f4a47c1f1756decc8f5d8d496f4cb074f4" Oct 30 20:31:05.691132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:05.691098 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:31:05.696691 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:05.696668 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d2720-6b57579f7b-2rj2n"] Oct 30 20:31:06.677821 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:06.677784 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" path="/var/lib/kubelet/pods/963d4ab2-b668-41bc-9ba8-7114f8f38a15/volumes" Oct 30 20:31:08.098295 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:08.098243 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:08.185499 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:08.185462 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:31:08.185911 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:08.185875 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" containerID="cri-o://a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88" gracePeriod=30 Oct 30 20:31:10.113206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:10.113170 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:13.445505 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.445477 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:31:13.580611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580580 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580611 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580622 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580655 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580720 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580742 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580775 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.580870 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580840 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.581163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580883 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.581163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580979 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\") pod \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\" (UID: \"5c07ee87-58d0-4abe-8f10-8781cdba3f77\") " Oct 30 20:31:13.581163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.580999 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.581163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581088 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:13.581418 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581261 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.581496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581473 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.581569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581495 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.581569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581510 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.581569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581520 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.581569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.581555 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.583324 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.583294 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "kube-api-access-nkp2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:13.583474 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.583411 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:13.583550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.583527 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:13.583661 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.583579 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy" pod "5c07ee87-58d0-4abe-8f10-8781cdba3f77" (UID: "5c07ee87-58d0-4abe-8f10-8781cdba3f77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.665680 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.665643 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 20:31:13.670810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.670781 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 20:31:13.682888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682855 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.682888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682889 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.683092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682903 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.683092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682916 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.683092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682929 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkp2p\" (UniqueName: \"kubernetes.io/projected/5c07ee87-58d0-4abe-8f10-8781cdba3f77-kube-api-access-nkp2p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.683092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.682943 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07ee87-58d0-4abe-8f10-8781cdba3f77-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.693830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.693751 5991 generic.go:356] "Generic (PLEG): container finished" podID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerID="a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88" exitCode=0 Oct 30 20:31:13.693830 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.693793 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" Oct 30 20:31:13.694025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.693834 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" event={"ID":"5c07ee87-58d0-4abe-8f10-8781cdba3f77","Type":"ContainerDied","Data":"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88"} Oct 30 20:31:13.694025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.693954 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6" event={"ID":"5c07ee87-58d0-4abe-8f10-8781cdba3f77","Type":"ContainerDied","Data":"2954c2653ffc46ebc107c6c227f7e63eda22848b3b0b5792fca1180b76af9b5c"} Oct 30 20:31:13.694025 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.693988 5991 scope.go:122] "RemoveContainer" containerID="a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88" Oct 30 20:31:13.704836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.704817 5991 scope.go:122] "RemoveContainer" containerID="a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88" Oct 30 20:31:13.705125 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:13.705098 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88\": container with ID starting with a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88 not found: ID does not exist" containerID="a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88" Oct 30 20:31:13.705237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.705133 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88"} err="failed to get container status \"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88\": rpc error: code = NotFound desc = could not find container \"a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88\": container with ID starting with a7ff9bf8cbc09a4f795668fa047e405f8478c0122439b772f192f9873a6f7a88 not found: ID does not exist" Oct 30 20:31:13.743622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.743589 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:31:13.745159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:13.745130 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-b7bx6"] Oct 30 20:31:14.678748 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:14.678715 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" path="/var/lib/kubelet/pods/5c07ee87-58d0-4abe-8f10-8781cdba3f77/volumes" Oct 30 20:31:18.097990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:18.097955 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:19.062711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:19.062672 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 20:31:20.113091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:20.113052 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:20.869136 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:20.869098 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:31:21.144659 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:21.144566 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:31:23.473416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:23.473374 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 20:31:24.063472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.063243 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" containerID="cri-o://5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357" gracePeriod=25 Oct 30 20:31:24.159003 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.158977 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:31:24.191093 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.191063 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:31:24.296621 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.296582 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\" (UniqueName: \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\") pod \"18bb5424-e691-4875-b046-f56eeffe32d4\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " Oct 30 20:31:24.296810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.296655 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\") pod \"6d2253ed-47bd-4dfa-8340-6974b1130605\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " Oct 30 20:31:24.296810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.296687 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\" (UniqueName: \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\") pod \"6d2253ed-47bd-4dfa-8340-6974b1130605\" (UID: \"6d2253ed-47bd-4dfa-8340-6974b1130605\") " Oct 30 20:31:24.296810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.296735 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\") pod \"18bb5424-e691-4875-b046-f56eeffe32d4\" (UID: \"18bb5424-e691-4875-b046-f56eeffe32d4\") " Oct 30 20:31:24.297128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.297103 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume" pod "18bb5424-e691-4875-b046-f56eeffe32d4" (UID: "18bb5424-e691-4875-b046-f56eeffe32d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:24.297185 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.297105 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume" pod "6d2253ed-47bd-4dfa-8340-6974b1130605" (UID: "6d2253ed-47bd-4dfa-8340-6974b1130605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:24.298842 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.298820 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2" pod "18bb5424-e691-4875-b046-f56eeffe32d4" (UID: "18bb5424-e691-4875-b046-f56eeffe32d4"). InnerVolumeSpecName "kube-api-access-4nts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:24.298979 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.298957 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z" pod "6d2253ed-47bd-4dfa-8340-6974b1130605" (UID: "6d2253ed-47bd-4dfa-8340-6974b1130605"). InnerVolumeSpecName "kube-api-access-c2x2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:24.398335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.398247 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bb5424-e691-4875-b046-f56eeffe32d4-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.398335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.398279 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nts2\" (UniqueName: \"kubernetes.io/projected/18bb5424-e691-4875-b046-f56eeffe32d4-kube-api-access-4nts2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.398335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.398288 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2253ed-47bd-4dfa-8340-6974b1130605-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.398335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.398296 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2x2z\" (UniqueName: \"kubernetes.io/projected/6d2253ed-47bd-4dfa-8340-6974b1130605-kube-api-access-c2x2z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.754219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.754183 5991 generic.go:356] "Generic (PLEG): container finished" podID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerID="b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915" exitCode=0 Oct 30 20:31:24.754734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.754241 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" Oct 30 20:31:24.754734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.754266 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" event={"ID":"6d2253ed-47bd-4dfa-8340-6974b1130605","Type":"ContainerDied","Data":"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915"} Oct 30 20:31:24.754734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.754322 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l" event={"ID":"6d2253ed-47bd-4dfa-8340-6974b1130605","Type":"ContainerDied","Data":"3ba3db9c1d3e3e590ee3b3e28fe4942757e7c1fd13b9f623b43fcde04f1b9272"} Oct 30 20:31:24.754734 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.754344 5991 scope.go:122] "RemoveContainer" containerID="b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915" Oct 30 20:31:24.755961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.755934 5991 generic.go:356] "Generic (PLEG): container finished" podID="18bb5424-e691-4875-b046-f56eeffe32d4" containerID="5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff" exitCode=0 Oct 30 20:31:24.756073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.755978 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" event={"ID":"18bb5424-e691-4875-b046-f56eeffe32d4","Type":"ContainerDied","Data":"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff"} Oct 30 20:31:24.756073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.756037 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" event={"ID":"18bb5424-e691-4875-b046-f56eeffe32d4","Type":"ContainerDied","Data":"c3fcb852700d4fb772f636fa3d636b33e626f9a435d4a3f0479785a86a027ec3"} Oct 30 20:31:24.756073 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.755991 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p" Oct 30 20:31:24.764086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.764067 5991 scope.go:122] "RemoveContainer" containerID="b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915" Oct 30 20:31:24.764388 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:24.764354 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915\": container with ID starting with b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915 not found: ID does not exist" containerID="b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915" Oct 30 20:31:24.764465 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.764397 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915"} err="failed to get container status \"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915\": rpc error: code = NotFound desc = could not find container \"b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915\": container with ID starting with b2223e5d5f27f6cc4823a3d8d52db3459c9f328f30b75efbf597169bcbcc6915 not found: ID does not exist" Oct 30 20:31:24.764465 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.764415 5991 scope.go:122] "RemoveContainer" containerID="5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff" Oct 30 20:31:24.772851 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.772833 5991 scope.go:122] "RemoveContainer" containerID="5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff" Oct 30 20:31:24.773113 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:24.773097 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff\": container with ID starting with 5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff not found: ID does not exist" containerID="5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff" Oct 30 20:31:24.773165 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.773120 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff"} err="failed to get container status \"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff\": rpc error: code = NotFound desc = could not find container \"5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff\": container with ID starting with 5863e4c17f3ff6afd407f3453a99d1991845bce6fd39875aa76aa1cde2dd70ff not found: ID does not exist" Oct 30 20:31:24.804578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.804545 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:31:24.816806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.816777 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5c875d-5fbdfd5594-ktb7p"] Oct 30 20:31:24.841600 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.841566 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:31:24.844894 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:24.844867 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5c875d-7d7d5c47d7-grg7l"] Oct 30 20:31:26.677507 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:26.677474 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" path="/var/lib/kubelet/pods/18bb5424-e691-4875-b046-f56eeffe32d4/volumes" Oct 30 20:31:26.677942 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:26.677886 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" path="/var/lib/kubelet/pods/6d2253ed-47bd-4dfa-8340-6974b1130605/volumes" Oct 30 20:31:28.474570 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.474486 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" containerID="cri-o://bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49" gracePeriod=25 Oct 30 20:31:28.687099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.687041 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" containerID="cri-o://efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb" gracePeriod=45 Oct 30 20:31:28.691626 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.691324 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" containerID="cri-o://faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837" gracePeriod=45 Oct 30 20:31:28.729648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.729563 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:28.955711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.955683 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 20:31:28.981211 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:28.981147 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 20:31:29.148992 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.148955 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.148997 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149043 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149071 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149101 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149136 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149160 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149195 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149182 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149237 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149291 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\" (UniqueName: \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\") pod \"4e7c0848-8746-483b-8ffb-f58640119bf8\" (UID: \"4e7c0848-8746-483b-8ffb-f58640119bf8\") " Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149322 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\" (UniqueName: \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149348 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149355 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.149596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149440 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\") pod \"deb22ca8-145d-4116-85bf-0ac9f44379e6\" (UID: \"deb22ca8-145d-4116-85bf-0ac9f44379e6\") " Oct 30 20:31:29.149927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149623 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.149927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149768 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.150041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149960 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.150041 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.149978 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.150265 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150242 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.150345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150269 5991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.150345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150285 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.150345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150299 5991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.150345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150310 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.150345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.150322 5991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.151753 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.151712 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "kube-api-access-49ffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:29.151856 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.151802 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.151927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.151876 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:29.152210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.152190 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "kube-api-access-tqm2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:29.152292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.152268 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs" pod "deb22ca8-145d-4116-85bf-0ac9f44379e6" (UID: "deb22ca8-145d-4116-85bf-0ac9f44379e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:29.152406 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.152359 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs" pod "4e7c0848-8746-483b-8ffb-f58640119bf8" (UID: "4e7c0848-8746-483b-8ffb-f58640119bf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.251937 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb22ca8-145d-4116-85bf-0ac9f44379e6-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.251965 5991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c0848-8746-483b-8ffb-f58640119bf8-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.251974 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c0848-8746-483b-8ffb-f58640119bf8-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.251984 5991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb22ca8-145d-4116-85bf-0ac9f44379e6-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.251993 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqm2k\" (UniqueName: \"kubernetes.io/projected/4e7c0848-8746-483b-8ffb-f58640119bf8-kube-api-access-tqm2k\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.252022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.252002 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ffc\" (UniqueName: \"kubernetes.io/projected/deb22ca8-145d-4116-85bf-0ac9f44379e6-kube-api-access-49ffc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.736521 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.736483 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:29.786763 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.786730 5991 generic.go:356] "Generic (PLEG): container finished" podID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerID="faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837" exitCode=0 Oct 30 20:31:29.786939 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.786779 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" Oct 30 20:31:29.786939 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.786812 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" event={"ID":"4e7c0848-8746-483b-8ffb-f58640119bf8","Type":"ContainerDied","Data":"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837"} Oct 30 20:31:29.786939 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.786859 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl" event={"ID":"4e7c0848-8746-483b-8ffb-f58640119bf8","Type":"ContainerDied","Data":"f88e45d0724e5d2012e5b89ae2aa41e3f6388c7d86c6f29664dd1242114f46f3"} Oct 30 20:31:29.786939 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.786883 5991 scope.go:122] "RemoveContainer" containerID="faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837" Oct 30 20:31:29.788243 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.788222 5991 generic.go:356] "Generic (PLEG): container finished" podID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerID="efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb" exitCode=0 Oct 30 20:31:29.788325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.788262 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" Oct 30 20:31:29.788325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.788255 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" event={"ID":"deb22ca8-145d-4116-85bf-0ac9f44379e6","Type":"ContainerDied","Data":"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb"} Oct 30 20:31:29.788325 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.788307 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms" event={"ID":"deb22ca8-145d-4116-85bf-0ac9f44379e6","Type":"ContainerDied","Data":"891247321d0e1620670d98545c50a49c69e77e0617853d23104cdcc369347880"} Oct 30 20:31:29.800207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.800159 5991 scope.go:122] "RemoveContainer" containerID="faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837" Oct 30 20:31:29.800530 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:29.800508 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837\": container with ID starting with faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837 not found: ID does not exist" containerID="faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837" Oct 30 20:31:29.800638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.800535 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837"} err="failed to get container status \"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837\": rpc error: code = NotFound desc = could not find container \"faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837\": container with ID starting with faf1e3682d95cdf85290b988832f8963faf8683a006ff51473a3122f2299e837 not found: ID does not exist" Oct 30 20:31:29.800638 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.800553 5991 scope.go:122] "RemoveContainer" containerID="efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb" Oct 30 20:31:29.808685 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.808670 5991 scope.go:122] "RemoveContainer" containerID="efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb" Oct 30 20:31:29.808923 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:29.808907 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb\": container with ID starting with efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb not found: ID does not exist" containerID="efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb" Oct 30 20:31:29.808980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.808931 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb"} err="failed to get container status \"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb\": rpc error: code = NotFound desc = could not find container \"efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb\": container with ID starting with efafeabde4f50efa6cad1c55677d05a4199b1ba34badec5d1a2622a7ed6d9ffb not found: ID does not exist" Oct 30 20:31:29.852709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.852679 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 20:31:29.864444 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.864417 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/premium-w0-5db436-kserve-7dcbb8ddb8-59gms"] Oct 30 20:31:29.898284 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.898256 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 20:31:29.899697 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:29.899674 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/simulator-w0-5db436-kserve-6467b4f8bf-tqswl"] Oct 30 20:31:30.677045 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:30.677012 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" path="/var/lib/kubelet/pods/4e7c0848-8746-483b-8ffb-f58640119bf8/volumes" Oct 30 20:31:30.677455 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:30.677439 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" path="/var/lib/kubelet/pods/deb22ca8-145d-4116-85bf-0ac9f44379e6/volumes" Oct 30 20:31:34.879043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:34.879015 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 20:31:34.879522 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:34.879249 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" podUID="a0050383-e123-49fe-be8f-2583d1e0b036" containerName="maas-api" containerID="cri-o://e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f" gracePeriod=30 Oct 30 20:31:35.148552 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.148525 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 20:31:35.211482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.211450 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\" (UniqueName: \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\") pod \"a0050383-e123-49fe-be8f-2583d1e0b036\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " Oct 30 20:31:35.211648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.211553 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\") pod \"a0050383-e123-49fe-be8f-2583d1e0b036\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " Oct 30 20:31:35.211701 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.211650 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\") pod \"a0050383-e123-49fe-be8f-2583d1e0b036\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " Oct 30 20:31:35.211738 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.211727 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\") pod \"a0050383-e123-49fe-be8f-2583d1e0b036\" (UID: \"a0050383-e123-49fe-be8f-2583d1e0b036\") " Oct 30 20:31:35.211972 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.211951 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca" pod "a0050383-e123-49fe-be8f-2583d1e0b036" (UID: "a0050383-e123-49fe-be8f-2583d1e0b036"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:35.212096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.212078 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0050383-e123-49fe-be8f-2583d1e0b036-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:35.213805 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.213763 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs" pod "a0050383-e123-49fe-be8f-2583d1e0b036" (UID: "a0050383-e123-49fe-be8f-2583d1e0b036"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:35.213956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.213833 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls" pod "a0050383-e123-49fe-be8f-2583d1e0b036" (UID: "a0050383-e123-49fe-be8f-2583d1e0b036"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:35.213956 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.213833 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn" pod "a0050383-e123-49fe-be8f-2583d1e0b036" (UID: "a0050383-e123-49fe-be8f-2583d1e0b036"). InnerVolumeSpecName "kube-api-access-5xmnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:35.312887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.312851 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:35.312887 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.312889 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmnn\" (UniqueName: \"kubernetes.io/projected/a0050383-e123-49fe-be8f-2583d1e0b036-kube-api-access-5xmnn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:35.313147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.312905 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0050383-e123-49fe-be8f-2583d1e0b036-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:35.833463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.833426 5991 generic.go:356] "Generic (PLEG): container finished" podID="a0050383-e123-49fe-be8f-2583d1e0b036" containerID="e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f" exitCode=0 Oct 30 20:31:35.833667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.833471 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" Oct 30 20:31:35.833667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.833504 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" event={"ID":"a0050383-e123-49fe-be8f-2583d1e0b036","Type":"ContainerDied","Data":"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f"} Oct 30 20:31:35.833667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.833554 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g" event={"ID":"a0050383-e123-49fe-be8f-2583d1e0b036","Type":"ContainerDied","Data":"c81d8a1f4a004df4e8e93f2bb699de6c3ab97336dd4823d35a08a63145695896"} Oct 30 20:31:35.833667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.833584 5991 scope.go:122] "RemoveContainer" containerID="e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f" Oct 30 20:31:35.843906 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.843884 5991 scope.go:122] "RemoveContainer" containerID="e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f" Oct 30 20:31:35.844232 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:35.844210 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f\": container with ID starting with e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f not found: ID does not exist" containerID="e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f" Oct 30 20:31:35.844303 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.844242 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f"} err="failed to get container status \"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f\": rpc error: code = NotFound desc = could not find container \"e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f\": container with ID starting with e3a656890f371a69e50dad95d1e8c6de8895804623061ac59df2b4f6f0fba37f not found: ID does not exist" Oct 30 20:31:35.886657 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.886574 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 20:31:35.899316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:35.899281 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5db436-86db9f74bc-9tc2g"] Oct 30 20:31:36.676917 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:36.676883 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0050383-e123-49fe-be8f-2583d1e0b036" path="/var/lib/kubelet/pods/a0050383-e123-49fe-be8f-2583d1e0b036/volumes" Oct 30 20:31:38.729711 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:38.729671 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:39.528126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:39.528079 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:31:39.736314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:39.736270 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:44.327459 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.327392 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 20:31:44.410075 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.410042 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\") pod \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " Oct 30 20:31:44.410272 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.410176 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\" (UniqueName: \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\") pod \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\" (UID: \"798aec19-6a8e-4bdf-81a6-d35e2a6717e6\") " Oct 30 20:31:44.410435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.410411 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume" pod "798aec19-6a8e-4bdf-81a6-d35e2a6717e6" (UID: "798aec19-6a8e-4bdf-81a6-d35e2a6717e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:44.412234 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.412208 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx" pod "798aec19-6a8e-4bdf-81a6-d35e2a6717e6" (UID: "798aec19-6a8e-4bdf-81a6-d35e2a6717e6"). InnerVolumeSpecName "kube-api-access-2kghx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:44.511709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.511670 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kghx\" (UniqueName: \"kubernetes.io/projected/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-kube-api-access-2kghx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:44.511709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.511700 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798aec19-6a8e-4bdf-81a6-d35e2a6717e6-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:44.894720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.894684 5991 generic.go:356] "Generic (PLEG): container finished" podID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerID="5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357" exitCode=0 Oct 30 20:31:44.894915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.894736 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" Oct 30 20:31:44.894915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.894754 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" event={"ID":"798aec19-6a8e-4bdf-81a6-d35e2a6717e6","Type":"ContainerDied","Data":"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357"} Oct 30 20:31:44.894915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.894797 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv" event={"ID":"798aec19-6a8e-4bdf-81a6-d35e2a6717e6","Type":"ContainerDied","Data":"162acc657c4ffc3542eddae1c60d290758377ff55b4d18514f55f9996335403c"} Oct 30 20:31:44.894915 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.894817 5991 scope.go:122] "RemoveContainer" containerID="5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357" Oct 30 20:31:44.903838 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.903813 5991 scope.go:122] "RemoveContainer" containerID="5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357" Oct 30 20:31:44.904150 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:44.904131 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357\": container with ID starting with 5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357 not found: ID does not exist" containerID="5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357" Oct 30 20:31:44.904202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.904161 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357"} err="failed to get container status \"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357\": rpc error: code = NotFound desc = could not find container \"5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357\": container with ID starting with 5ff842e9f9376254149878cc6300991f3a6bc2defd5f48c57cbdddbb2137e357 not found: ID does not exist" Oct 30 20:31:44.992204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.992168 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 20:31:44.999715 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:44.999682 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5db436-5b77bd565f-z7jfv"] Oct 30 20:31:46.676709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:46.676676 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" path="/var/lib/kubelet/pods/798aec19-6a8e-4bdf-81a6-d35e2a6717e6/volumes" Oct 30 20:31:48.730258 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.730190 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 20:31:48.855605 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.855563 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\") pod \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " Oct 30 20:31:48.855822 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.855627 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\" (UniqueName: \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\") pod \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\" (UID: \"2e306930-f287-4d59-ad46-9ae4eaeb4f50\") " Oct 30 20:31:48.855973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.855946 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume" pod "2e306930-f287-4d59-ad46-9ae4eaeb4f50" (UID: "2e306930-f287-4d59-ad46-9ae4eaeb4f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:48.857900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.857873 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn" pod "2e306930-f287-4d59-ad46-9ae4eaeb4f50" (UID: "2e306930-f287-4d59-ad46-9ae4eaeb4f50"). InnerVolumeSpecName "kube-api-access-hjrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:48.917069 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.917032 5991 generic.go:356] "Generic (PLEG): container finished" podID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerID="bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49" exitCode=0 Oct 30 20:31:48.917289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.917094 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" Oct 30 20:31:48.917289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.917126 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" event={"ID":"2e306930-f287-4d59-ad46-9ae4eaeb4f50","Type":"ContainerDied","Data":"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49"} Oct 30 20:31:48.917289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.917216 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4" event={"ID":"2e306930-f287-4d59-ad46-9ae4eaeb4f50","Type":"ContainerDied","Data":"ed33723febaa8d40b9c17375b2433e512a24d4d4952f0e0c85cc2812de6d87a0"} Oct 30 20:31:48.917289 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.917249 5991 scope.go:122] "RemoveContainer" containerID="bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49" Oct 30 20:31:48.927074 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.927051 5991 scope.go:122] "RemoveContainer" containerID="bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49" Oct 30 20:31:48.927440 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:48.927418 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49\": container with ID starting with bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49 not found: ID does not exist" containerID="bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49" Oct 30 20:31:48.927488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.927451 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49"} err="failed to get container status \"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49\": rpc error: code = NotFound desc = could not find container \"bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49\": container with ID starting with bf27dac53c6e5efe00f9f5b30e4048c249750b1e39c65511dbb95658dec93a49 not found: ID does not exist" Oct 30 20:31:48.957093 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.957052 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e306930-f287-4d59-ad46-9ae4eaeb4f50-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:48.957093 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.957089 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrfn\" (UniqueName: \"kubernetes.io/projected/2e306930-f287-4d59-ad46-9ae4eaeb4f50-kube-api-access-hjrfn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:48.970597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.970555 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 20:31:48.974991 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:48.974951 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-109cfe-c468d456-6xgs4"] Oct 30 20:31:49.271026 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.270986 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 20:31:49.271248 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.271177 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" podUID="e811c39f-15d9-4189-a386-b3c8db802672" containerName="maas-api" containerID="cri-o://32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a" gracePeriod=30 Oct 30 20:31:49.539251 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.539165 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 20:31:49.664103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.664064 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\") pod \"e811c39f-15d9-4189-a386-b3c8db802672\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " Oct 30 20:31:49.664103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.664116 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\") pod \"e811c39f-15d9-4189-a386-b3c8db802672\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " Oct 30 20:31:49.664375 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.664151 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\") pod \"e811c39f-15d9-4189-a386-b3c8db802672\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " Oct 30 20:31:49.664375 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.664180 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\" (UniqueName: \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\") pod \"e811c39f-15d9-4189-a386-b3c8db802672\" (UID: \"e811c39f-15d9-4189-a386-b3c8db802672\") " Oct 30 20:31:49.664686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.664650 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca" pod "e811c39f-15d9-4189-a386-b3c8db802672" (UID: "e811c39f-15d9-4189-a386-b3c8db802672"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:49.666577 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.666542 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d" pod "e811c39f-15d9-4189-a386-b3c8db802672" (UID: "e811c39f-15d9-4189-a386-b3c8db802672"). InnerVolumeSpecName "kube-api-access-7rn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:49.666721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.666628 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls" pod "e811c39f-15d9-4189-a386-b3c8db802672" (UID: "e811c39f-15d9-4189-a386-b3c8db802672"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:49.666721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.666638 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs" pod "e811c39f-15d9-4189-a386-b3c8db802672" (UID: "e811c39f-15d9-4189-a386-b3c8db802672"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:49.766296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.766260 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.766296 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.766295 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e811c39f-15d9-4189-a386-b3c8db802672-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.766769 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.766309 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e811c39f-15d9-4189-a386-b3c8db802672-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.766769 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.766319 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rn8d\" (UniqueName: \"kubernetes.io/projected/e811c39f-15d9-4189-a386-b3c8db802672-kube-api-access-7rn8d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.925122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.925023 5991 generic.go:356] "Generic (PLEG): container finished" podID="e811c39f-15d9-4189-a386-b3c8db802672" containerID="32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a" exitCode=0 Oct 30 20:31:49.925122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.925075 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" event={"ID":"e811c39f-15d9-4189-a386-b3c8db802672","Type":"ContainerDied","Data":"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a"} Oct 30 20:31:49.925122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.925078 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" Oct 30 20:31:49.925122 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.925118 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw" event={"ID":"e811c39f-15d9-4189-a386-b3c8db802672","Type":"ContainerDied","Data":"4d019a92cba4aab2000eed874d0084d8e6d06389c2d4c2c10a32abcb9f79df7b"} Oct 30 20:31:49.925549 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.925140 5991 scope.go:122] "RemoveContainer" containerID="32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a" Oct 30 20:31:49.935297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.935275 5991 scope.go:122] "RemoveContainer" containerID="32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a" Oct 30 20:31:49.935724 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:49.935698 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a\": container with ID starting with 32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a not found: ID does not exist" containerID="32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a" Oct 30 20:31:49.935835 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.935732 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a"} err="failed to get container status \"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a\": rpc error: code = NotFound desc = could not find container \"32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a\": container with ID starting with 32cbdfd4d755f113ffb2584645d41bbba75fb3aadf5695add7b6f94319afe24a not found: ID does not exist" Oct 30 20:31:49.975516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.975480 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 20:31:49.987631 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:49.987593 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f349a1-7df48d9ddf-l2qpw"] Oct 30 20:31:50.679696 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:50.679664 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" path="/var/lib/kubelet/pods/2e306930-f287-4d59-ad46-9ae4eaeb4f50/volumes" Oct 30 20:31:50.680154 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:50.680137 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e811c39f-15d9-4189-a386-b3c8db802672" path="/var/lib/kubelet/pods/e811c39f-15d9-4189-a386-b3c8db802672/volumes" Oct 30 20:31:50.869709 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:31:50.869669 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:31:54.177238 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:31:54.177198 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:32:02.281900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:02.281861 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:32:20.869196 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:32:20.869158 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:32:27.227962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:27.227908 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:32:45.494193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494110 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk"] Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494567 5991 cpu_manager.go:414] "containerMap: removing container" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494579 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494590 5991 cpu_manager.go:414] "containerMap: removing container" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494595 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494607 5991 cpu_manager.go:414] "containerMap: removing container" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" Oct 30 20:32:45.494612 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494612 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494623 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494628 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494636 5991 cpu_manager.go:414] "containerMap: removing container" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494641 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494648 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494653 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494663 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494667 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494673 5991 cpu_manager.go:414] "containerMap: removing container" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494677 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494684 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494689 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494694 5991 cpu_manager.go:414] "containerMap: removing container" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494698 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494705 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b97b046d-1071-4817-8c94-403ecb2ed589" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494709 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97b046d-1071-4817-8c94-403ecb2ed589" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494716 5991 cpu_manager.go:414] "containerMap: removing container" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494720 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494729 5991 cpu_manager.go:414] "containerMap: removing container" podUID="3b25294d-fe92-4a3e-becc-db8ef0331beb" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494733 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b25294d-fe92-4a3e-becc-db8ef0331beb" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494739 5991 cpu_manager.go:414] "containerMap: removing container" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494743 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494751 5991 cpu_manager.go:414] "containerMap: removing container" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494755 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494764 5991 cpu_manager.go:414] "containerMap: removing container" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494768 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494776 5991 cpu_manager.go:414] "containerMap: removing container" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494780 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494786 5991 cpu_manager.go:414] "containerMap: removing container" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494790 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494801 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a0050383-e123-49fe-be8f-2583d1e0b036" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494806 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0050383-e123-49fe-be8f-2583d1e0b036" containerName="maas-api" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494816 5991 cpu_manager.go:414] "containerMap: removing container" podUID="7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" containerName="revoke-keys" Oct 30 20:32:45.494806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494821 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" containerName="revoke-keys" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494829 5991 cpu_manager.go:414] "containerMap: removing container" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494833 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494845 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e811c39f-15d9-4189-a386-b3c8db802672" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494849 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e811c39f-15d9-4189-a386-b3c8db802672" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494858 5991 cpu_manager.go:414] "containerMap: removing container" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494862 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494868 5991 cpu_manager.go:414] "containerMap: removing container" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494872 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494878 5991 cpu_manager.go:414] "containerMap: removing container" podUID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494882 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494890 5991 cpu_manager.go:414] "containerMap: removing container" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494894 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494950 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b25294d-fe92-4a3e-becc-db8ef0331beb" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494956 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bb5424-e691-4875-b046-f56eeffe32d4" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494961 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494968 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="798aec19-6a8e-4bdf-81a6-d35e2a6717e6" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494974 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb22ca8-145d-4116-85bf-0ac9f44379e6" containerName="main" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494981 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494985 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2253ed-47bd-4dfa-8340-6974b1130605" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494990 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c6216b-e0b5-4858-8bc7-72793593fa13" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.494997 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0050383-e123-49fe-be8f-2583d1e0b036" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495002 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" containerName="revoke-keys" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495006 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02f4a3f-4d82-4688-8b80-83b7d2332378" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495011 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e811c39f-15d9-4189-a386-b3c8db802672" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495018 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cc3275-277b-4536-8ae6-7db2e9126f37" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495022 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="901efb89-7a25-40ea-9fb5-8d3f95286f72" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495028 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="963d4ab2-b668-41bc-9ba8-7114f8f38a15" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495033 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6207f6-765e-4cb5-a8de-83106ab16872" containerName="main" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495040 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07ee87-58d0-4abe-8f10-8781cdba3f77" containerName="istio-proxy" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495045 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7c0848-8746-483b-8ffb-f58640119bf8" containerName="main" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495050 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f006e551-4480-4475-a270-1d803ab8259e" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495055 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e306930-f287-4d59-ad46-9ae4eaeb4f50" containerName="payload-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495060 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495065 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495071 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c78352-08a5-4222-a2e0-5849f9ae8fcd" containerName="payload-pre-processing" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495075 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33cfc0d-ddbf-40df-bb30-e636a9a3aee2" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495080 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495085 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97b046d-1071-4817-8c94-403ecb2ed589" containerName="maas-api" Oct 30 20:32:45.495881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.495089 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee65e032-5d15-4462-ac7f-300575e6fc4d" containerName="istio-proxy" Oct 30 20:32:45.498306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.498280 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.526879 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.526845 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk"] Oct 30 20:32:45.550253 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550176 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550301 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550335 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84p5\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-kube-api-access-c84p5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550483 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550419 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550526 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550573 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550618 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550694 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.550953 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.550724 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.651850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651791 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.651850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651861 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651894 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651929 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651953 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.651986 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652016 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652047 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84p5\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-kube-api-access-c84p5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652076 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652288 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652314 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652480 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652725 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652599 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.652802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.652777 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.654334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.654315 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.654497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.654477 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.671349 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.671316 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.671782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.671752 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84p5\" (UniqueName: \"kubernetes.io/projected/5386bfdc-a1aa-40ec-b5de-2481f63e5184-kube-api-access-c84p5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-77bmk\" (UID: \"5386bfdc-a1aa-40ec-b5de-2481f63e5184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:45.811862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:45.811761 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:46.127564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.127525 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk"] Oct 30 20:32:46.128773 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:32:46.128748 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5386bfdc_a1aa_40ec_b5de_2481f63e5184.slice/crio-a7bc351f29a2675a4b1d3c5ea12b5458def2600a65956714036d36978b66d04d WatchSource:0}: Error finding container a7bc351f29a2675a4b1d3c5ea12b5458def2600a65956714036d36978b66d04d: Status 404 returned error can't find the container with id a7bc351f29a2675a4b1d3c5ea12b5458def2600a65956714036d36978b66d04d Oct 30 20:32:46.131677 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.131642 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:46.131752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.131737 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:46.131802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.131780 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:46.244553 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.244517 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" event={"ID":"5386bfdc-a1aa-40ec-b5de-2481f63e5184","Type":"ContainerStarted","Data":"7b55aa57eaa635d938b3fe0e46fe4f3a5d162033e439ffa663af0935fb785e80"} Oct 30 20:32:46.244743 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.244570 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" event={"ID":"5386bfdc-a1aa-40ec-b5de-2481f63e5184","Type":"ContainerStarted","Data":"a7bc351f29a2675a4b1d3c5ea12b5458def2600a65956714036d36978b66d04d"} Oct 30 20:32:46.289167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.289094 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podStartSLOduration=1.289073667 podStartE2EDuration="1.289073667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:46.287782377 +0000 UTC m=+3433.737275101" watchObservedRunningTime="2026-10-30 20:32:46.289073667 +0000 UTC m=+3433.738566393" Oct 30 20:32:46.812313 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:46.812263 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:47.813739 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:47.813698 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:47.814265 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:47.813766 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:48.812173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:48.812133 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:48.812407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:48.812206 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:49.812958 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:49.812918 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:49.813415 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:49.812985 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:50.812438 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:50.812399 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:50.812672 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:50.812470 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:50.892805 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:32:50.892766 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:32:51.813139 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:51.813100 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:51.813342 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:51.813166 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:52.813204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:52.813164 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 20:32:52.813716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:52.813229 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 20:32:54.516278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:54.516228 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:54.516846 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:54.516820 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:32:54.517543 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:32:54.517521 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" Oct 30 20:33:18.022759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:18.022721 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:33:20.871381 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:33:20.871317 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:33:34.989645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:34.989613 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:33:39.312334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:39.312296 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:33:45.684810 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.684769 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj"] Oct 30 20:33:45.685267 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.685252 5991 cpu_manager.go:414] "containerMap: removing container" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:33:45.685314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.685268 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b156438c-2e22-43d0-accb-e534fa7843ed" containerName="cleanup" Oct 30 20:33:45.685314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.685280 5991 cpu_manager.go:414] "containerMap: removing container" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:33:45.685314 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.685285 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:33:45.685454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.685385 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bea4eb-4c43-4dcb-bd45-48e9e0123a28" containerName="cleanup" Oct 30 20:33:45.688732 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.688709 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.718283 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.718246 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj"] Oct 30 20:33:45.818051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818011 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818051 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818056 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94a0aec-c628-48a5-bd9c-8483df0281fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818186 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818237 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hl5\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-kube-api-access-v4hl5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818297 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818257 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818351 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818399 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818484 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818460 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.818605 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.818496 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919113 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919155 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94a0aec-c628-48a5-bd9c-8483df0281fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919188 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919211 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hl5\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-kube-api-access-v4hl5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919231 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919281 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919308 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919332 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919470 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919408 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919873 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919791 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919892 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.919938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919910 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.920016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.919930 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.920094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.920073 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94a0aec-c628-48a5-bd9c-8483df0281fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.921560 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.921538 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.921862 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.921846 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.938753 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.938701 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:45.939016 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:45.938992 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hl5\" (UniqueName: \"kubernetes.io/projected/f94a0aec-c628-48a5-bd9c-8483df0281fb-kube-api-access-v4hl5\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rgnvj\" (UID: \"f94a0aec-c628-48a5-bd9c-8483df0281fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:46.006276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.006241 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:46.316341 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.316308 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj"] Oct 30 20:33:46.317312 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:33:46.317284 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94a0aec_c628_48a5_bd9c_8483df0281fb.slice/crio-77a5a39fb76fecb678f96d5f4225a031aee60a65e3f42f0bf36bbb6b6c3e4f25 WatchSource:0}: Error finding container 77a5a39fb76fecb678f96d5f4225a031aee60a65e3f42f0bf36bbb6b6c3e4f25: Status 404 returned error can't find the container with id 77a5a39fb76fecb678f96d5f4225a031aee60a65e3f42f0bf36bbb6b6c3e4f25 Oct 30 20:33:46.319503 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.319471 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:46.319606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.319547 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:46.319606 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.319595 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:46.592577 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.592481 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" event={"ID":"f94a0aec-c628-48a5-bd9c-8483df0281fb","Type":"ContainerStarted","Data":"7a2081e27eb0d037112cf86d843f8d1e6bea80973cbef73d918c24d55d23a450"} Oct 30 20:33:46.592577 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.592531 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" event={"ID":"f94a0aec-c628-48a5-bd9c-8483df0281fb","Type":"ContainerStarted","Data":"77a5a39fb76fecb678f96d5f4225a031aee60a65e3f42f0bf36bbb6b6c3e4f25"} Oct 30 20:33:46.636567 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:46.636513 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podStartSLOduration=1.636497428 podStartE2EDuration="1.636497428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:46.635766277 +0000 UTC m=+3494.085259003" watchObservedRunningTime="2026-10-30 20:33:46.636497428 +0000 UTC m=+3494.085990153" Oct 30 20:33:47.006731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:47.006692 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:47.007986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:47.007959 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:47.008111 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:47.008007 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:48.006709 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:48.006670 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:48.006898 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:48.006746 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:49.007343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:49.007306 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:49.007782 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:49.007398 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:50.006737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:50.006699 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:50.006927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:50.006766 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:50.892431 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:33:50.892393 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:33:51.007346 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:51.007313 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:51.007559 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:51.007422 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:52.007092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:52.007051 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:52.007548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:52.007111 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:53.007210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:53.007169 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 20:33:53.007665 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:53.007240 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 20:33:54.548678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:54.548641 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:54.549163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:54.549049 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:54.549988 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:54.549965 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" Oct 30 20:33:55.913298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:33:55.913254 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:02.002107 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:02.002068 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:10.399926 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:10.399886 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:20.867441 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:34:20.867399 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:34:33.686216 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:33.686179 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:36.038601 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:36.038558 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:34:36.039235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:36.039197 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:34:43.995975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:43.995929 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:48.392629 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:48.392580 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:34:50.942660 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:34:50.942609 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:34:51.032391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:51.031537 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-psrq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:34:51.032391 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:34:51.031611 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-psrq4" podUID="c9fda10e-bbc9-4010-b06d-a2c2d7427f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:05.576237 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:05.576199 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:35:10.457286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:10.457240 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:35:10.589392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:10.581537 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-77bmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.589392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:10.581645 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-77bmk" podUID="5386bfdc-a1aa-40ec-b5de-2481f63e5184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:10.598388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:10.596594 5991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rgnvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.598388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:10.596671 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rgnvj" podUID="f94a0aec-c628-48a5-bd9c-8483df0281fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:16.035356 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:16.035305 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp"] Oct 30 20:35:16.047146 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:16.047108 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d9a785-bfhbp"] Oct 30 20:35:16.679630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:16.679582 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef60147-b9a6-4e10-a979-ba5cbbcc4e92" path="/var/lib/kubelet/pods/7ef60147-b9a6-4e10-a979-ba5cbbcc4e92/volumes" Oct 30 20:35:20.869797 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:35:20.869758 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:35:27.297005 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:27.296963 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:35:33.092984 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:33.092936 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:35:33.337804 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:33.337769 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:35:33.363733 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:33.363652 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:35:48.392349 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:48.392262 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:35:50.868581 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:35:50.868540 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:35:53.268396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:35:53.263797 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:08.317730 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:08.317694 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:19.182022 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:19.181979 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:20.938805 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:36:20.938723 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:36:24.214907 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:24.214869 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:35.069284 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:35.069248 5991 scope.go:122] "RemoveContainer" containerID="95ba085ca496c8e52c3c69f2fa7463a1e9925790a1a455a7c104c60cf8773493" Oct 30 20:36:41.216807 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:41.216762 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:46.279315 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:46.279276 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:36:50.868439 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:36:50.868396 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:36:56.116695 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:36:56.116657 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:37:14.216534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:14.216447 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:37:20.869085 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:37:20.869039 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:37:24.533745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:24.533710 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:37:44.435335 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:44.435275 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:37:50.189034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:50.188993 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:37:50.869259 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:37:50.869215 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:37:59.873923 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:59.873886 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:37:59.877963 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:59.877938 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:37:59.902517 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:59.902481 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:37:59.908981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:37:59.908950 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-26e5b0-serving-cert\"" Oct 30 20:38:00.006397 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.006343 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.006595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.006471 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrx7\" (UniqueName: \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.006595 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.006532 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.006678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.006610 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.107254 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.107213 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.107520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.107282 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrx7\" (UniqueName: \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.107520 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.107311 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.107520 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:38:00.107417 5991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-serving-cert: secret "maas-api-e2e-worker-single-main-26e5b0-serving-cert" not found Oct 30 20:38:00.107520 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:38:00.107508 5991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls podName:45ffefec-a972-497f-bb63-bd690923c5bf nodeName:}" failed. No retries permitted until 2026-10-30 20:38:00.607486477 +0000 UTC m=+3748.056979184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls") pod "maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" (UID: "45ffefec-a972-497f-bb63-bd690923c5bf") : secret "maas-api-e2e-worker-single-main-26e5b0-serving-cert" not found Oct 30 20:38:00.107783 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.107431 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.108131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.108107 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.109880 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.109860 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.134336 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.134260 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrx7\" (UniqueName: \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.357418 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.357384 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:38:00.361511 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.361475 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.379902 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.379872 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4bf82de9-dockercfg-756zm\"" Oct 30 20:38:00.381736 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.381706 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:38:00.411597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.411511 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.411597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.411561 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gnx\" (UniqueName: \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.433818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.433789 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-26e5b0\"" Oct 30 20:38:00.513098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.513060 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.513306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.513127 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gnx\" (UniqueName: \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.513828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.513806 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.533164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.533129 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gnx\" (UniqueName: \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\") pod \"payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.614688 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.614652 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.617162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.617132 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.675488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.675416 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:00.791918 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.791877 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:00.976731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:00.976697 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:38:00.977949 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:38:00.977924 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65baaf60_d6ca_43c1_8908_2d8d6b19fb09.slice/crio-bfcacc7849675b3e17ec228f7dbdfc1fe4f4defeed715c23090d876d125cac4b WatchSource:0}: Error finding container bfcacc7849675b3e17ec228f7dbdfc1fe4f4defeed715c23090d876d125cac4b: Status 404 returned error can't find the container with id bfcacc7849675b3e17ec228f7dbdfc1fe4f4defeed715c23090d876d125cac4b Oct 30 20:38:01.096177 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.096139 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:38:01.096927 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.096896 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" event={"ID":"65baaf60-d6ca-43c1-8908-2d8d6b19fb09","Type":"ContainerStarted","Data":"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a"} Oct 30 20:38:01.097034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.096954 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" event={"ID":"65baaf60-d6ca-43c1-8908-2d8d6b19fb09","Type":"ContainerStarted","Data":"bfcacc7849675b3e17ec228f7dbdfc1fe4f4defeed715c23090d876d125cac4b"} Oct 30 20:38:01.097104 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.097087 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:01.097810 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:38:01.097789 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ffefec_a972_497f_bb63_bd690923c5bf.slice/crio-5a06f588fe277d838e1954d839ac625a797aaa2c608d8e7ef76a4d0eb47adad0 WatchSource:0}: Error finding container 5a06f588fe277d838e1954d839ac625a797aaa2c608d8e7ef76a4d0eb47adad0: Status 404 returned error can't find the container with id 5a06f588fe277d838e1954d839ac625a797aaa2c608d8e7ef76a4d0eb47adad0 Oct 30 20:38:01.099180 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.099131 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:38:01.141547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:01.141472 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" podStartSLOduration=1.141453914 podStartE2EDuration="1.141453914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:01.139752654 +0000 UTC m=+3748.589245378" watchObservedRunningTime="2026-10-30 20:38:01.141453914 +0000 UTC m=+3748.590946654" Oct 30 20:38:02.103352 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:02.103317 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" event={"ID":"45ffefec-a972-497f-bb63-bd690923c5bf","Type":"ContainerStarted","Data":"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14"} Oct 30 20:38:02.103352 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:02.103392 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" event={"ID":"45ffefec-a972-497f-bb63-bd690923c5bf","Type":"ContainerStarted","Data":"5a06f588fe277d838e1954d839ac625a797aaa2c608d8e7ef76a4d0eb47adad0"} Oct 30 20:38:02.103977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:02.103585 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:02.155435 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:02.155338 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" podStartSLOduration=2.752481798 podStartE2EDuration="3.155317332s" totalImagesPullingTime="402.835534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:02.153215525 +0000 UTC m=+3749.602708250" watchObservedRunningTime="2026-10-30 20:38:02.155317332 +0000 UTC m=+3749.604810092" Oct 30 20:38:03.531156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:03.531111 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:38:08.114991 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:08.114961 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:38:12.107308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:12.107274 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:38:17.253951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:17.253908 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:38:20.868768 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:38:20.868724 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:38:33.812306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:33.812268 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:38:50.869759 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:38:50.869668 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:38:57.810387 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:38:57.810317 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:39:06.917836 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:06.917792 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:39:09.389843 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.389804 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:39:09.390355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.390025 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" podUID="45ffefec-a972-497f-bb63-bd690923c5bf" containerName="maas-api" containerID="cri-o://cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14" gracePeriod=30 Oct 30 20:39:09.857305 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.857275 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:39:09.960319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.960229 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\" (UniqueName: \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\") pod \"45ffefec-a972-497f-bb63-bd690923c5bf\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " Oct 30 20:39:09.960319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.960290 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") pod \"45ffefec-a972-497f-bb63-bd690923c5bf\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " Oct 30 20:39:09.960547 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.960488 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\") pod \"45ffefec-a972-497f-bb63-bd690923c5bf\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " Oct 30 20:39:09.960591 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.960545 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\") pod \"45ffefec-a972-497f-bb63-bd690923c5bf\" (UID: \"45ffefec-a972-497f-bb63-bd690923c5bf\") " Oct 30 20:39:09.960978 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.960954 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca" pod "45ffefec-a972-497f-bb63-bd690923c5bf" (UID: "45ffefec-a972-497f-bb63-bd690923c5bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:09.962538 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.962510 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls" pod "45ffefec-a972-497f-bb63-bd690923c5bf" (UID: "45ffefec-a972-497f-bb63-bd690923c5bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:09.962632 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.962533 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs" pod "45ffefec-a972-497f-bb63-bd690923c5bf" (UID: "45ffefec-a972-497f-bb63-bd690923c5bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:09.962632 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:09.962511 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7" pod "45ffefec-a972-497f-bb63-bd690923c5bf" (UID: "45ffefec-a972-497f-bb63-bd690923c5bf"). InnerVolumeSpecName "kube-api-access-frrx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:10.062481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.062439 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:10.062481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.062473 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ffefec-a972-497f-bb63-bd690923c5bf-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:10.062481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.062482 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ffefec-a972-497f-bb63-bd690923c5bf-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:10.062481 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.062493 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrx7\" (UniqueName: \"kubernetes.io/projected/45ffefec-a972-497f-bb63-bd690923c5bf-kube-api-access-frrx7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:10.473309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.473268 5991 generic.go:356] "Generic (PLEG): container finished" podID="45ffefec-a972-497f-bb63-bd690923c5bf" containerID="cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14" exitCode=0 Oct 30 20:39:10.473793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.473314 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" Oct 30 20:39:10.473793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.473350 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" event={"ID":"45ffefec-a972-497f-bb63-bd690923c5bf","Type":"ContainerDied","Data":"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14"} Oct 30 20:39:10.473793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.473438 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg" event={"ID":"45ffefec-a972-497f-bb63-bd690923c5bf","Type":"ContainerDied","Data":"5a06f588fe277d838e1954d839ac625a797aaa2c608d8e7ef76a4d0eb47adad0"} Oct 30 20:39:10.473793 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.473464 5991 scope.go:122] "RemoveContainer" containerID="cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14" Oct 30 20:39:10.493324 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.493293 5991 scope.go:122] "RemoveContainer" containerID="cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14" Oct 30 20:39:10.493699 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:39:10.493679 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14\": container with ID starting with cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14 not found: ID does not exist" containerID="cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14" Oct 30 20:39:10.493746 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.493709 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14"} err="failed to get container status \"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14\": rpc error: code = NotFound desc = could not find container \"cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14\": container with ID starting with cae49af6eeb4fcaafe71345657b434786c55f571f672000b4aa8fbc71dc23a14 not found: ID does not exist" Oct 30 20:39:10.532226 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.532190 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:39:10.542159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.542120 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26e5b0-94664c5cd-546hg"] Oct 30 20:39:10.677233 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:10.677197 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ffefec-a972-497f-bb63-bd690923c5bf" path="/var/lib/kubelet/pods/45ffefec-a972-497f-bb63-bd690923c5bf/volumes" Oct 30 20:39:11.712482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:11.712440 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:39:11.919048 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:11.919001 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" containerID="cri-o://cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a" gracePeriod=25 Oct 30 20:39:12.105132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:12.105090 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:39:20.868610 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:39:20.868566 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:39:22.105286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:22.105246 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:39:30.081562 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:30.081523 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:39:32.104134 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:39:32.104057 5991 logging.go:55] [core] [Channel #13636 SubChannel #13637]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.197:50052", ServerName: "10.134.0.197:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.197:50052: connect: connection refused" Oct 30 20:39:32.246791 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.246758 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:39:32.340345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.340308 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\") pod \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " Oct 30 20:39:32.340345 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.340354 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\" (UniqueName: \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\") pod \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\" (UID: \"65baaf60-d6ca-43c1-8908-2d8d6b19fb09\") " Oct 30 20:39:32.340750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.340727 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume" pod "65baaf60-d6ca-43c1-8908-2d8d6b19fb09" (UID: "65baaf60-d6ca-43c1-8908-2d8d6b19fb09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:32.342684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.342662 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx" pod "65baaf60-d6ca-43c1-8908-2d8d6b19fb09" (UID: "65baaf60-d6ca-43c1-8908-2d8d6b19fb09"). InnerVolumeSpecName "kube-api-access-98gnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:32.442047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.442005 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:32.442047 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.442039 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98gnx\" (UniqueName: \"kubernetes.io/projected/65baaf60-d6ca-43c1-8908-2d8d6b19fb09-kube-api-access-98gnx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:39:32.625287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.625202 5991 generic.go:356] "Generic (PLEG): container finished" podID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerID="cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a" exitCode=0 Oct 30 20:39:32.625287 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.625255 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" Oct 30 20:39:32.625584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.625262 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" event={"ID":"65baaf60-d6ca-43c1-8908-2d8d6b19fb09","Type":"ContainerDied","Data":"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a"} Oct 30 20:39:32.625584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.625421 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" event={"ID":"65baaf60-d6ca-43c1-8908-2d8d6b19fb09","Type":"ContainerDied","Data":"bfcacc7849675b3e17ec228f7dbdfc1fe4f4defeed715c23090d876d125cac4b"} Oct 30 20:39:32.625584 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.625459 5991 scope.go:122] "RemoveContainer" containerID="cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a" Oct 30 20:39:32.634813 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.634790 5991 scope.go:122] "RemoveContainer" containerID="cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a" Oct 30 20:39:32.635103 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:39:32.635084 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a\": container with ID starting with cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a not found: ID does not exist" containerID="cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a" Oct 30 20:39:32.635162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.635115 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a"} err="failed to get container status \"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a\": rpc error: code = NotFound desc = could not find container \"cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a\": container with ID starting with cf24b071b375d5dc408a881a0ba7b137e8a2d3186dd004b5098609486469107a not found: ID does not exist" Oct 30 20:39:32.677155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.677111 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:39:32.682144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:32.682118 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6"] Oct 30 20:39:33.103937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:33.103893 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26e5b0-6b7477xnw6" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.197:50052\" within 1s: context deadline exceeded" Oct 30 20:39:34.676716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:34.676683 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" path="/var/lib/kubelet/pods/65baaf60-d6ca-43c1-8908-2d8d6b19fb09/volumes" Oct 30 20:39:35.598116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:39:35.598050 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:39:50.868425 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:39:50.868384 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:40:20.871061 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:40:20.870979 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:40:33.372523 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:40:33.372489 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:40:33.399043 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:40:33.399014 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:40:50.868538 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:40:50.868498 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:41:02.982827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:41:02.982765 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:41:18.312679 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:41:18.312644 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:41:20.876065 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:41:20.876020 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:41:23.183648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:41:23.183609 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:41:50.867349 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:41:50.867268 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:42:20.868389 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:42:20.868329 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:42:32.180925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:32.180881 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:42:39.496891 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.496488 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.496997 5991 cpu_manager.go:414] "containerMap: removing container" podUID="45ffefec-a972-497f-bb63-bd690923c5bf" containerName="maas-api" Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.497011 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ffefec-a972-497f-bb63-bd690923c5bf" containerName="maas-api" Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.497059 5991 cpu_manager.go:414] "containerMap: removing container" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.497066 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.497143 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="65baaf60-d6ca-43c1-8908-2d8d6b19fb09" containerName="payload-pre-processing" Oct 30 20:42:39.497396 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.497154 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ffefec-a972-497f-bb63-bd690923c5bf" containerName="maas-api" Oct 30 20:42:39.500752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.500733 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.520869 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.520827 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-051802-serving-cert\"" Oct 30 20:42:39.530256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.530221 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:42:39.603686 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.603653 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68rb\" (UniqueName: \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.603881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.603721 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.603881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.603801 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.603881 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.603833 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.705290 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.705253 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.705516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.705299 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.705516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.705402 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68rb\" (UniqueName: \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.705516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.705466 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.706156 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.706125 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.708231 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.708200 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.708597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.708575 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.723637 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.723605 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68rb\" (UniqueName: \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\") pod \"maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.814476 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.814388 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:39.952641 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.952599 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:42:39.957019 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.956986 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:39.976480 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.976448 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-051802\"" Oct 30 20:42:39.978727 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:39.978678 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:42:40.008934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.008886 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.009161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.008951 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wd6m\" (UniqueName: \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.110163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.110064 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.110163 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.110142 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wd6m\" (UniqueName: \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.111500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.111461 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.130975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.130932 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wd6m\" (UniqueName: \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\") pod \"payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.133712 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.133684 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:42:40.135761 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:42:40.135736 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42819699_276e_4411_9675_d7b47c5db514.slice/crio-d20c3881f840cbb1aa64c9cc300214fa4fcf93aeb62574440fb582871ffcd90a WatchSource:0}: Error finding container d20c3881f840cbb1aa64c9cc300214fa4fcf93aeb62574440fb582871ffcd90a: Status 404 returned error can't find the container with id d20c3881f840cbb1aa64c9cc300214fa4fcf93aeb62574440fb582871ffcd90a Oct 30 20:42:40.271827 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.271788 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.578596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.578309 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:42:40.581126 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:42:40.581090 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55b78be_a38f_4f29_86b7_4f2482cf42fe.slice/crio-c3eb6971fdbf933edca6c3712bf7309d87d6fcac5254dac79275262bffec7379 WatchSource:0}: Error finding container c3eb6971fdbf933edca6c3712bf7309d87d6fcac5254dac79275262bffec7379: Status 404 returned error can't find the container with id c3eb6971fdbf933edca6c3712bf7309d87d6fcac5254dac79275262bffec7379 Oct 30 20:42:40.649938 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.649901 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" event={"ID":"e55b78be-a38f-4f29-86b7-4f2482cf42fe","Type":"ContainerStarted","Data":"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75"} Oct 30 20:42:40.650140 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.649971 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:40.650140 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.649999 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" event={"ID":"e55b78be-a38f-4f29-86b7-4f2482cf42fe","Type":"ContainerStarted","Data":"c3eb6971fdbf933edca6c3712bf7309d87d6fcac5254dac79275262bffec7379"} Oct 30 20:42:40.651700 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.651673 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" event={"ID":"42819699-276e-4411-9675-d7b47c5db514","Type":"ContainerStarted","Data":"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08"} Oct 30 20:42:40.651833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.651717 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" event={"ID":"42819699-276e-4411-9675-d7b47c5db514","Type":"ContainerStarted","Data":"d20c3881f840cbb1aa64c9cc300214fa4fcf93aeb62574440fb582871ffcd90a"} Oct 30 20:42:40.651833 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.651740 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:40.701973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.701906 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" podStartSLOduration=1.70188774 podStartE2EDuration="1.70188774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:40.700265744 +0000 UTC m=+4028.149758471" watchObservedRunningTime="2026-10-30 20:42:40.70188774 +0000 UTC m=+4028.151380466" Oct 30 20:42:40.740162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:40.740102 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" podStartSLOduration=1.35658766 podStartE2EDuration="1.74008733s" totalImagesPullingTime="383.49967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:40.739224397 +0000 UTC m=+4028.188717146" watchObservedRunningTime="2026-10-30 20:42:40.74008733 +0000 UTC m=+4028.189580054" Oct 30 20:42:42.916962 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:42.916927 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:42:46.664106 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:46.664071 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:42:50.871325 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:42:50.871281 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:42:51.661205 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:51.661173 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:42:52.624740 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:42:52.624701 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:43:20.869920 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:43:20.869830 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:43:49.778545 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:43:49.778504 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:43:50.869257 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:43:50.869213 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:44:13.789806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:13.789772 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:44:20.867919 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:44:20.867880 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:44:23.571309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:23.571270 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:44:24.341196 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.341153 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:44:24.341426 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.341386 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" podUID="42819699-276e-4411-9675-d7b47c5db514" containerName="maas-api" containerID="cri-o://19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08" gracePeriod=30 Oct 30 20:44:24.608801 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.608741 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:44:24.713951 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.713921 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\") pod \"42819699-276e-4411-9675-d7b47c5db514\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " Oct 30 20:44:24.714124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.714038 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\" (UniqueName: \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\") pod \"42819699-276e-4411-9675-d7b47c5db514\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " Oct 30 20:44:24.714124 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.714076 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\") pod \"42819699-276e-4411-9675-d7b47c5db514\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " Oct 30 20:44:24.714252 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.714122 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\") pod \"42819699-276e-4411-9675-d7b47c5db514\" (UID: \"42819699-276e-4411-9675-d7b47c5db514\") " Oct 30 20:44:24.714316 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.714249 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca" pod "42819699-276e-4411-9675-d7b47c5db514" (UID: "42819699-276e-4411-9675-d7b47c5db514"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:24.714443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.714426 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42819699-276e-4411-9675-d7b47c5db514-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:24.716170 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.716147 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls" pod "42819699-276e-4411-9675-d7b47c5db514" (UID: "42819699-276e-4411-9675-d7b47c5db514"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:24.716260 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.716171 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb" pod "42819699-276e-4411-9675-d7b47c5db514" (UID: "42819699-276e-4411-9675-d7b47c5db514"). InnerVolumeSpecName "kube-api-access-q68rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:24.716302 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.716263 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs" pod "42819699-276e-4411-9675-d7b47c5db514" (UID: "42819699-276e-4411-9675-d7b47c5db514"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:24.815847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.815808 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68rb\" (UniqueName: \"kubernetes.io/projected/42819699-276e-4411-9675-d7b47c5db514-kube-api-access-q68rb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:24.817010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.816981 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:24.817010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:24.817009 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42819699-276e-4411-9675-d7b47c5db514-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:25.201250 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.201206 5991 generic.go:356] "Generic (PLEG): container finished" podID="42819699-276e-4411-9675-d7b47c5db514" containerID="19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08" exitCode=0 Oct 30 20:44:25.201488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.201397 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" event={"ID":"42819699-276e-4411-9675-d7b47c5db514","Type":"ContainerDied","Data":"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08"} Oct 30 20:44:25.201488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.201453 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" Oct 30 20:44:25.201488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.201478 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw" event={"ID":"42819699-276e-4411-9675-d7b47c5db514","Type":"ContainerDied","Data":"d20c3881f840cbb1aa64c9cc300214fa4fcf93aeb62574440fb582871ffcd90a"} Oct 30 20:44:25.201703 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.201507 5991 scope.go:122] "RemoveContainer" containerID="19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08" Oct 30 20:44:25.213597 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.213575 5991 scope.go:122] "RemoveContainer" containerID="19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08" Oct 30 20:44:25.216037 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:44:25.215655 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08\": container with ID starting with 19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08 not found: ID does not exist" containerID="19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08" Oct 30 20:44:25.216037 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.215695 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08"} err="failed to get container status \"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08\": rpc error: code = NotFound desc = could not find container \"19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08\": container with ID starting with 19faf97d3092baf6cd453283e714ffbd1a8678029bfb0adb8f129a8618a07b08 not found: ID does not exist" Oct 30 20:44:25.248210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.248180 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:44:25.258278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:25.258254 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-051802-8db7d788b-v8mqw"] Oct 30 20:44:26.677034 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:26.676999 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42819699-276e-4411-9675-d7b47c5db514" path="/var/lib/kubelet/pods/42819699-276e-4411-9675-d7b47c5db514/volumes" Oct 30 20:44:26.833868 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:26.833834 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:44:28.572042 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:28.571974 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" containerID="cri-o://f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75" gracePeriod=25 Oct 30 20:44:31.658895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:31.658853 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:41.333359 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:41.333316 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:44:41.659076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:41.658990 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:48.821749 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:48.821719 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:44:48.900099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:48.899996 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\" (UniqueName: \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\") pod \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " Oct 30 20:44:48.900099 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:48.900060 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\") pod \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\" (UID: \"e55b78be-a38f-4f29-86b7-4f2482cf42fe\") " Oct 30 20:44:48.900588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:48.900554 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume" pod "e55b78be-a38f-4f29-86b7-4f2482cf42fe" (UID: "e55b78be-a38f-4f29-86b7-4f2482cf42fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:48.902450 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:48.902425 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m" pod "e55b78be-a38f-4f29-86b7-4f2482cf42fe" (UID: "e55b78be-a38f-4f29-86b7-4f2482cf42fe"). InnerVolumeSpecName "kube-api-access-6wd6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:49.001202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.001140 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wd6m\" (UniqueName: \"kubernetes.io/projected/e55b78be-a38f-4f29-86b7-4f2482cf42fe-kube-api-access-6wd6m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:49.001202 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.001194 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55b78be-a38f-4f29-86b7-4f2482cf42fe-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:44:49.367731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.367666 5991 generic.go:356] "Generic (PLEG): container finished" podID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerID="f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75" exitCode=0 Oct 30 20:44:49.367952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.367736 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" event={"ID":"e55b78be-a38f-4f29-86b7-4f2482cf42fe","Type":"ContainerDied","Data":"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75"} Oct 30 20:44:49.367952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.367795 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" Oct 30 20:44:49.367952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.367822 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9" event={"ID":"e55b78be-a38f-4f29-86b7-4f2482cf42fe","Type":"ContainerDied","Data":"c3eb6971fdbf933edca6c3712bf7309d87d6fcac5254dac79275262bffec7379"} Oct 30 20:44:49.367952 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.367846 5991 scope.go:122] "RemoveContainer" containerID="f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75" Oct 30 20:44:49.378098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.378072 5991 scope.go:122] "RemoveContainer" containerID="f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75" Oct 30 20:44:49.378495 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:44:49.378469 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75\": container with ID starting with f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75 not found: ID does not exist" containerID="f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75" Oct 30 20:44:49.378601 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.378506 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75"} err="failed to get container status \"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75\": rpc error: code = NotFound desc = could not find container \"f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75\": container with ID starting with f58fdbca817e28589a70807dd02e1e30ebe0e5e8cd7cad33b6042f7f770e3c75 not found: ID does not exist" Oct 30 20:44:49.420539 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.420503 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:44:49.430602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:49.430563 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-051802-568b9fdf6c46p9"] Oct 30 20:44:50.677608 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:50.677574 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" path="/var/lib/kubelet/pods/e55b78be-a38f-4f29-86b7-4f2482cf42fe/volumes" Oct 30 20:44:50.870355 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:44:50.870310 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:44:58.630540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:44:58.630499 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:13.823324 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:13.823288 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:20.869623 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:45:20.869580 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:45:25.619091 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:25.619052 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:33.408094 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:33.408065 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:45:33.443624 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:33.443530 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:45:33.879506 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:33.879469 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:44.388925 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:44.388874 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:49.636390 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:45:49.635400 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:45:50.868761 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:45:50.868712 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:46:04.809961 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:04.809920 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:46:06.087729 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.087683 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq"] Oct 30 20:46:06.088550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088524 5991 cpu_manager.go:414] "containerMap: removing container" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" Oct 30 20:46:06.088550 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088547 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" Oct 30 20:46:06.088712 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088576 5991 cpu_manager.go:414] "containerMap: removing container" podUID="42819699-276e-4411-9675-d7b47c5db514" containerName="maas-api" Oct 30 20:46:06.088712 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088581 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42819699-276e-4411-9675-d7b47c5db514" containerName="maas-api" Oct 30 20:46:06.088712 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088649 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="42819699-276e-4411-9675-d7b47c5db514" containerName="maas-api" Oct 30 20:46:06.088712 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.088663 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55b78be-a38f-4f29-86b7-4f2482cf42fe" containerName="payload-processing" Oct 30 20:46:06.092318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.092297 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.119424 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.119336 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq"] Oct 30 20:46:06.240888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.240837 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjdv\" (UniqueName: \"kubernetes.io/projected/5e90c01f-09e5-4dff-a6a4-2a25e056db24-kube-api-access-bqjdv\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.240888 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.240899 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-metrics-certs\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.241168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.241053 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-maas-api-tls\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.241168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.241074 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e90c01f-09e5-4dff-a6a4-2a25e056db24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.341767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.341678 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjdv\" (UniqueName: \"kubernetes.io/projected/5e90c01f-09e5-4dff-a6a4-2a25e056db24-kube-api-access-bqjdv\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.341767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.341721 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-metrics-certs\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.341995 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.341787 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-maas-api-tls\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.341995 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.341809 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e90c01f-09e5-4dff-a6a4-2a25e056db24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.342645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.342620 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e90c01f-09e5-4dff-a6a4-2a25e056db24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.344415 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.344389 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-maas-api-tls\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.344651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.344632 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e90c01f-09e5-4dff-a6a4-2a25e056db24-metrics-certs\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.361159 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.361120 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjdv\" (UniqueName: \"kubernetes.io/projected/5e90c01f-09e5-4dff-a6a4-2a25e056db24-kube-api-access-bqjdv\") pod \"maas-api-76f6f44c6f-72rhq\" (UID: \"5e90c01f-09e5-4dff-a6a4-2a25e056db24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.409360 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.409319 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:06.709308 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.709203 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq"] Oct 30 20:46:06.712551 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:06.712513 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e90c01f_09e5_4dff_a6a4_2a25e056db24.slice/crio-9c446fcec7563067305c4b8688f22bfde3362f8fee18e54c7808d3e77453e90a WatchSource:0}: Error finding container 9c446fcec7563067305c4b8688f22bfde3362f8fee18e54c7808d3e77453e90a: Status 404 returned error can't find the container with id 9c446fcec7563067305c4b8688f22bfde3362f8fee18e54c7808d3e77453e90a Oct 30 20:46:06.713867 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.713850 5991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:46:06.839660 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:06.839624 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" event={"ID":"5e90c01f-09e5-4dff-a6a4-2a25e056db24","Type":"ContainerStarted","Data":"9c446fcec7563067305c4b8688f22bfde3362f8fee18e54c7808d3e77453e90a"} Oct 30 20:46:07.845679 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:07.845639 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" event={"ID":"5e90c01f-09e5-4dff-a6a4-2a25e056db24","Type":"ContainerStarted","Data":"51eabde7e64f3784c295b43caf24535bcc3c7699d89493e8dfc89593bb2aebfc"} Oct 30 20:46:07.846113 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:07.845703 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:07.882135 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:07.882051 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" podStartSLOduration=1.550632022 podStartE2EDuration="1.882038769s" totalImagesPullingTime="331.406747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:07.881686133 +0000 UTC m=+4235.331178859" watchObservedRunningTime="2026-10-30 20:46:07.882038769 +0000 UTC m=+4235.331531491" Oct 30 20:46:13.856847 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:13.856763 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-72rhq" Oct 30 20:46:17.842578 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.842538 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:17.846589 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.846560 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:17.860548 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.860520 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f5c2161a\"" Oct 30 20:46:17.869086 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.869055 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f5c2161a-dockercfg-5cmsz\"" Oct 30 20:46:17.870755 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.870725 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:17.932648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.932612 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:17.936872 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.936845 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:17.956133 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.956100 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:17.968876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.968843 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqf4\" (UniqueName: \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:17.969070 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:17.968970 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.070534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.070498 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.070534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.070538 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nw5\" (UniqueName: \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.070759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.070701 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqf4\" (UniqueName: \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.070759 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.070740 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.071298 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.071277 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.088149 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.088115 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqf4\" (UniqueName: \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\") pod \"payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.161596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.161513 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.171746 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.171713 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.171910 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.171791 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nw5\" (UniqueName: \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.172630 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.172602 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.191133 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.191098 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nw5\" (UniqueName: \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\") pod \"payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.250829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.250790 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.461911 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.461888 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:18.464072 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:18.464041 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337c8f7c_722c_4430_94bc_eb6fbf8b89b9.slice/crio-32776a0771c13a5d1e0b047e6898607091b3e757a58ffdeaf787eee188cf0ca4 WatchSource:0}: Error finding container 32776a0771c13a5d1e0b047e6898607091b3e757a58ffdeaf787eee188cf0ca4: Status 404 returned error can't find the container with id 32776a0771c13a5d1e0b047e6898607091b3e757a58ffdeaf787eee188cf0ca4 Oct 30 20:46:18.554504 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.554475 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:18.556885 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:18.556857 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fa6941_e570_4386_99da_729afec9999b.slice/crio-4fef572ed802eb9cdbe2a677e16ea0f360b4c624d655b107d2cb9add57329495 WatchSource:0}: Error finding container 4fef572ed802eb9cdbe2a677e16ea0f360b4c624d655b107d2cb9add57329495: Status 404 returned error can't find the container with id 4fef572ed802eb9cdbe2a677e16ea0f360b4c624d655b107d2cb9add57329495 Oct 30 20:46:18.827140 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.827095 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:18.831000 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.830975 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.862967 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.862927 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f5c2161a-serving-cert\"" Oct 30 20:46:18.868407 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.868376 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:18.878105 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.878067 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdst8\" (UniqueName: \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.878351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.878150 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.878351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.878241 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.878351 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.878282 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.903249 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.903209 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" event={"ID":"42fa6941-e570-4386-99da-729afec9999b","Type":"ContainerStarted","Data":"eeac748f3f81cb68adc4811cec7b0fe54cdcecae9ba3378b382765483a2f226b"} Oct 30 20:46:18.903461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.903279 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:18.903461 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.903307 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" event={"ID":"42fa6941-e570-4386-99da-729afec9999b","Type":"ContainerStarted","Data":"4fef572ed802eb9cdbe2a677e16ea0f360b4c624d655b107d2cb9add57329495"} Oct 30 20:46:18.905482 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.905448 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" event={"ID":"337c8f7c-722c-4430-94bc-eb6fbf8b89b9","Type":"ContainerStarted","Data":"b70ff4b8b24eb32319b219bf3e75bf623002dd6756ece558c10fed0d85f6e1db"} Oct 30 20:46:18.905644 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.905507 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" event={"ID":"337c8f7c-722c-4430-94bc-eb6fbf8b89b9","Type":"ContainerStarted","Data":"32776a0771c13a5d1e0b047e6898607091b3e757a58ffdeaf787eee188cf0ca4"} Oct 30 20:46:18.905644 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.905586 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:18.946975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.946916 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" podStartSLOduration=1.946899085 podStartE2EDuration="1.946899085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:18.945686835 +0000 UTC m=+4246.395179562" watchObservedRunningTime="2026-10-30 20:46:18.946899085 +0000 UTC m=+4246.396391803" Oct 30 20:46:18.978904 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.978861 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdst8\" (UniqueName: \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.979110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.978931 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.979110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.979015 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.979110 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.979083 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.979756 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.979728 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.981634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.981615 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.981710 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.981636 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:18.998262 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:18.998204 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" podStartSLOduration=1.998190711 podStartE2EDuration="1.998190711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:18.996844344 +0000 UTC m=+4246.446337069" watchObservedRunningTime="2026-10-30 20:46:18.998190711 +0000 UTC m=+4246.447683449" Oct 30 20:46:19.003818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:19.003792 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdst8\" (UniqueName: \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\") pod \"maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:19.145575 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:19.145472 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:19.472323 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:19.472283 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:19.473105 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:19.473076 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234ef00_f5bc_424f_9ec2_499f63e0c217.slice/crio-5ef3a6b712f632fa3095c0048f24e3b1663c13bb0159214b1637e3621a3c2028 WatchSource:0}: Error finding container 5ef3a6b712f632fa3095c0048f24e3b1663c13bb0159214b1637e3621a3c2028: Status 404 returned error can't find the container with id 5ef3a6b712f632fa3095c0048f24e3b1663c13bb0159214b1637e3621a3c2028 Oct 30 20:46:19.911990 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:19.911594 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" event={"ID":"0234ef00-f5bc-424f-9ec2-499f63e0c217","Type":"ContainerStarted","Data":"5ef3a6b712f632fa3095c0048f24e3b1663c13bb0159214b1637e3621a3c2028"} Oct 30 20:46:20.869165 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:46:20.869126 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:46:20.918937 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:20.918901 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" event={"ID":"0234ef00-f5bc-424f-9ec2-499f63e0c217","Type":"ContainerStarted","Data":"21ab615ec5603a906e5b0153cc2c7fd3868ca469b692cb4949f8d06c62b4e742"} Oct 30 20:46:20.919377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:20.919004 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:20.965224 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:20.965166 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" podStartSLOduration=2.563115956 podStartE2EDuration="2.965150043s" totalImagesPullingTime="402.034087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:20.963916503 +0000 UTC m=+4248.413409229" watchObservedRunningTime="2026-10-30 20:46:20.965150043 +0000 UTC m=+4248.414642805" Oct 30 20:46:21.272531 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:21.272494 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 20:46:21.272795 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:21.272757 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" podUID="a1a7b7ab-3445-4878-ba81-277a85be52c4" containerName="maas-api" containerID="cri-o://d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd" gracePeriod=30 Oct 30 20:46:22.437794 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.437754 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:46:22.533392 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.533353 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 20:46:22.617312 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617220 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\") pod \"a1a7b7ab-3445-4878-ba81-277a85be52c4\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " Oct 30 20:46:22.617493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617372 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\" (UniqueName: \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\") pod \"a1a7b7ab-3445-4878-ba81-277a85be52c4\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " Oct 30 20:46:22.617493 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617464 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\") pod \"a1a7b7ab-3445-4878-ba81-277a85be52c4\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " Oct 30 20:46:22.617721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617533 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\") pod \"a1a7b7ab-3445-4878-ba81-277a85be52c4\" (UID: \"a1a7b7ab-3445-4878-ba81-277a85be52c4\") " Oct 30 20:46:22.617721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617664 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca" pod "a1a7b7ab-3445-4878-ba81-277a85be52c4" (UID: "a1a7b7ab-3445-4878-ba81-277a85be52c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:22.617995 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.617969 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7b7ab-3445-4878-ba81-277a85be52c4-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:22.619747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.619710 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs" pod "a1a7b7ab-3445-4878-ba81-277a85be52c4" (UID: "a1a7b7ab-3445-4878-ba81-277a85be52c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:22.619747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.619725 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls" pod "a1a7b7ab-3445-4878-ba81-277a85be52c4" (UID: "a1a7b7ab-3445-4878-ba81-277a85be52c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:22.619919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.619852 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq" pod "a1a7b7ab-3445-4878-ba81-277a85be52c4" (UID: "a1a7b7ab-3445-4878-ba81-277a85be52c4"). InnerVolumeSpecName "kube-api-access-f9nnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:22.719596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.719557 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9nnq\" (UniqueName: \"kubernetes.io/projected/a1a7b7ab-3445-4878-ba81-277a85be52c4-kube-api-access-f9nnq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:22.719596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.719594 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:22.719596 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.719609 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a7b7ab-3445-4878-ba81-277a85be52c4-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:22.930092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.930011 5991 generic.go:356] "Generic (PLEG): container finished" podID="a1a7b7ab-3445-4878-ba81-277a85be52c4" containerID="d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd" exitCode=0 Oct 30 20:46:22.930092 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.930061 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" Oct 30 20:46:22.930288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.930100 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" event={"ID":"a1a7b7ab-3445-4878-ba81-277a85be52c4","Type":"ContainerDied","Data":"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd"} Oct 30 20:46:22.930288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.930158 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-slr29" event={"ID":"a1a7b7ab-3445-4878-ba81-277a85be52c4","Type":"ContainerDied","Data":"7555d535ffe3febda60b21ba98c311c56e7b4f19c89827bbbd4c06b8988a334f"} Oct 30 20:46:22.930288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.930188 5991 scope.go:122] "RemoveContainer" containerID="d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd" Oct 30 20:46:22.939478 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.939455 5991 scope.go:122] "RemoveContainer" containerID="d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd" Oct 30 20:46:22.939798 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:46:22.939776 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd\": container with ID starting with d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd not found: ID does not exist" containerID="d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd" Oct 30 20:46:22.939865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.939810 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd"} err="failed to get container status \"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd\": rpc error: code = NotFound desc = could not find container \"d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd\": container with ID starting with d5def91e3138277289a538441371d1f7a162d25a12bcf56841431502c538dacd not found: ID does not exist" Oct 30 20:46:22.980056 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.980015 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 20:46:22.989610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:22.989576 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-slr29"] Oct 30 20:46:24.677496 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:24.677456 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a7b7ab-3445-4878-ba81-277a85be52c4" path="/var/lib/kubelet/pods/a1a7b7ab-3445-4878-ba81-277a85be52c4/volumes" Oct 30 20:46:26.929834 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:26.929805 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:29.915306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:29.915273 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:29.915768 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:29.915342 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:41.847376 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:41.847331 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:41.880658 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:41.880614 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:42.831745 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:42.831709 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:42.832309 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:42.832273 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" podUID="0234ef00-f5bc-424f-9ec2-499f63e0c217" containerName="maas-api" containerID="cri-o://21ab615ec5603a906e5b0153cc2c7fd3868ca469b692cb4949f8d06c62b4e742" gracePeriod=30 Oct 30 20:46:43.040936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.040902 5991 generic.go:356] "Generic (PLEG): container finished" podID="0234ef00-f5bc-424f-9ec2-499f63e0c217" containerID="21ab615ec5603a906e5b0153cc2c7fd3868ca469b692cb4949f8d06c62b4e742" exitCode=0 Oct 30 20:46:43.041327 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.040977 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" event={"ID":"0234ef00-f5bc-424f-9ec2-499f63e0c217","Type":"ContainerDied","Data":"21ab615ec5603a906e5b0153cc2c7fd3868ca469b692cb4949f8d06c62b4e742"} Oct 30 20:46:43.097246 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.097175 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:43.241776 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.241736 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\") pod \"0234ef00-f5bc-424f-9ec2-499f63e0c217\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " Oct 30 20:46:43.241969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.241819 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\" (UniqueName: \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\") pod \"0234ef00-f5bc-424f-9ec2-499f63e0c217\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " Oct 30 20:46:43.241969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.241873 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\") pod \"0234ef00-f5bc-424f-9ec2-499f63e0c217\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " Oct 30 20:46:43.241969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.241964 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\") pod \"0234ef00-f5bc-424f-9ec2-499f63e0c217\" (UID: \"0234ef00-f5bc-424f-9ec2-499f63e0c217\") " Oct 30 20:46:43.242206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.242180 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca" pod "0234ef00-f5bc-424f-9ec2-499f63e0c217" (UID: "0234ef00-f5bc-424f-9ec2-499f63e0c217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:43.242417 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.242401 5991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0234ef00-f5bc-424f-9ec2-499f63e0c217-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:43.244120 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.244095 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs" pod "0234ef00-f5bc-424f-9ec2-499f63e0c217" (UID: "0234ef00-f5bc-424f-9ec2-499f63e0c217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:43.244224 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.244141 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls" pod "0234ef00-f5bc-424f-9ec2-499f63e0c217" (UID: "0234ef00-f5bc-424f-9ec2-499f63e0c217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:43.244224 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.244172 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8" pod "0234ef00-f5bc-424f-9ec2-499f63e0c217" (UID: "0234ef00-f5bc-424f-9ec2-499f63e0c217"). InnerVolumeSpecName "kube-api-access-mdst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:43.343977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.343931 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdst8\" (UniqueName: \"kubernetes.io/projected/0234ef00-f5bc-424f-9ec2-499f63e0c217-kube-api-access-mdst8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:43.343977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.343972 5991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:43.343977 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:43.343982 5991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0234ef00-f5bc-424f-9ec2-499f63e0c217-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:44.048219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.048186 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" Oct 30 20:46:44.048219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.048209 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4" event={"ID":"0234ef00-f5bc-424f-9ec2-499f63e0c217","Type":"ContainerDied","Data":"5ef3a6b712f632fa3095c0048f24e3b1663c13bb0159214b1637e3621a3c2028"} Oct 30 20:46:44.048735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.048274 5991 scope.go:122] "RemoveContainer" containerID="21ab615ec5603a906e5b0153cc2c7fd3868ca469b692cb4949f8d06c62b4e742" Oct 30 20:46:44.092235 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.092190 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:44.101905 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.101869 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c2161a-8594496bd4-rbsp4"] Oct 30 20:46:44.677291 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:44.677258 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0234ef00-f5bc-424f-9ec2-499f63e0c217" path="/var/lib/kubelet/pods/0234ef00-f5bc-424f-9ec2-499f63e0c217/volumes" Oct 30 20:46:45.007684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:45.007643 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:46:46.848462 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:46.848410 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" podUID="42fa6941-e570-4386-99da-729afec9999b" containerName="payload-processing" containerID="cri-o://eeac748f3f81cb68adc4811cec7b0fe54cdcecae9ba3378b382765483a2f226b" gracePeriod=25 Oct 30 20:46:46.881153 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:46.881118 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" podUID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" containerName="payload-pre-processing" containerID="cri-o://b70ff4b8b24eb32319b219bf3e75bf623002dd6756ece558c10fed0d85f6e1db" gracePeriod=25 Oct 30 20:46:47.066924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.066885 5991 generic.go:356] "Generic (PLEG): container finished" podID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" containerID="b70ff4b8b24eb32319b219bf3e75bf623002dd6756ece558c10fed0d85f6e1db" exitCode=0 Oct 30 20:46:47.067233 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.066948 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" event={"ID":"337c8f7c-722c-4430-94bc-eb6fbf8b89b9","Type":"ContainerDied","Data":"b70ff4b8b24eb32319b219bf3e75bf623002dd6756ece558c10fed0d85f6e1db"} Oct 30 20:46:47.071924 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.071899 5991 generic.go:356] "Generic (PLEG): container finished" podID="42fa6941-e570-4386-99da-729afec9999b" containerID="eeac748f3f81cb68adc4811cec7b0fe54cdcecae9ba3378b382765483a2f226b" exitCode=0 Oct 30 20:46:47.072050 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.071970 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" event={"ID":"42fa6941-e570-4386-99da-729afec9999b","Type":"ContainerDied","Data":"eeac748f3f81cb68adc4811cec7b0fe54cdcecae9ba3378b382765483a2f226b"} Oct 30 20:46:47.137082 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.137053 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:47.166425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.166399 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:47.286659 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.286625 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\") pod \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " Oct 30 20:46:47.286841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.286696 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\") pod \"42fa6941-e570-4386-99da-729afec9999b\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " Oct 30 20:46:47.286841 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.286805 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\" (UniqueName: \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\") pod \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\" (UID: \"337c8f7c-722c-4430-94bc-eb6fbf8b89b9\") " Oct 30 20:46:47.286934 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.286839 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\" (UniqueName: \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\") pod \"42fa6941-e570-4386-99da-729afec9999b\" (UID: \"42fa6941-e570-4386-99da-729afec9999b\") " Oct 30 20:46:47.287052 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.287032 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume" pod "337c8f7c-722c-4430-94bc-eb6fbf8b89b9" (UID: "337c8f7c-722c-4430-94bc-eb6fbf8b89b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:47.287095 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.287071 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume" pod "42fa6941-e570-4386-99da-729afec9999b" (UID: "42fa6941-e570-4386-99da-729afec9999b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:47.287339 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.287318 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:47.287490 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.287341 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fa6941-e570-4386-99da-729afec9999b-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:47.289064 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.289041 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5" pod "42fa6941-e570-4386-99da-729afec9999b" (UID: "42fa6941-e570-4386-99da-729afec9999b"). InnerVolumeSpecName "kube-api-access-97nw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:47.289183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.289076 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4" pod "337c8f7c-722c-4430-94bc-eb6fbf8b89b9" (UID: "337c8f7c-722c-4430-94bc-eb6fbf8b89b9"). InnerVolumeSpecName "kube-api-access-9sqf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:47.388463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.388343 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqf4\" (UniqueName: \"kubernetes.io/projected/337c8f7c-722c-4430-94bc-eb6fbf8b89b9-kube-api-access-9sqf4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:47.388463 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:47.388393 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nw5\" (UniqueName: \"kubernetes.io/projected/42fa6941-e570-4386-99da-729afec9999b-kube-api-access-97nw5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:46:48.079564 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.079536 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" Oct 30 20:46:48.079993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.079567 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr" event={"ID":"337c8f7c-722c-4430-94bc-eb6fbf8b89b9","Type":"ContainerDied","Data":"32776a0771c13a5d1e0b047e6898607091b3e757a58ffdeaf787eee188cf0ca4"} Oct 30 20:46:48.079993 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.079638 5991 scope.go:122] "RemoveContainer" containerID="b70ff4b8b24eb32319b219bf3e75bf623002dd6756ece558c10fed0d85f6e1db" Oct 30 20:46:48.081162 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.081135 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" event={"ID":"42fa6941-e570-4386-99da-729afec9999b","Type":"ContainerDied","Data":"4fef572ed802eb9cdbe2a677e16ea0f360b4c624d655b107d2cb9add57329495"} Oct 30 20:46:48.081276 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.081166 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6" Oct 30 20:46:48.089155 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.089135 5991 scope.go:122] "RemoveContainer" containerID="eeac748f3f81cb68adc4811cec7b0fe54cdcecae9ba3378b382765483a2f226b" Oct 30 20:46:48.122802 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.122708 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:48.126999 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.126894 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5c2161a-7b5c85858d-t2hd6"] Oct 30 20:46:48.165114 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.165080 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:48.169619 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.169578 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5c2161a-5f597f6b47-lmtbr"] Oct 30 20:46:48.676605 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.676567 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" path="/var/lib/kubelet/pods/337c8f7c-722c-4430-94bc-eb6fbf8b89b9/volumes" Oct 30 20:46:48.676975 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:48.676962 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fa6941-e570-4386-99da-729afec9999b" path="/var/lib/kubelet/pods/42fa6941-e570-4386-99da-729afec9999b/volumes" Oct 30 20:46:49.907603 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.907562 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:46:49.908192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908163 5991 cpu_manager.go:414] "containerMap: removing container" podUID="42fa6941-e570-4386-99da-729afec9999b" containerName="payload-processing" Oct 30 20:46:49.908192 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908189 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fa6941-e570-4386-99da-729afec9999b" containerName="payload-processing" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908209 5991 cpu_manager.go:414] "containerMap: removing container" podUID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" containerName="payload-pre-processing" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908225 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" containerName="payload-pre-processing" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908239 5991 cpu_manager.go:414] "containerMap: removing container" podUID="a1a7b7ab-3445-4878-ba81-277a85be52c4" containerName="maas-api" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908246 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a7b7ab-3445-4878-ba81-277a85be52c4" containerName="maas-api" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908268 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0234ef00-f5bc-424f-9ec2-499f63e0c217" containerName="maas-api" Oct 30 20:46:49.908411 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908275 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0234ef00-f5bc-424f-9ec2-499f63e0c217" containerName="maas-api" Oct 30 20:46:49.908620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908458 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fa6941-e570-4386-99da-729afec9999b" containerName="payload-processing" Oct 30 20:46:49.908620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908470 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="337c8f7c-722c-4430-94bc-eb6fbf8b89b9" containerName="payload-pre-processing" Oct 30 20:46:49.908620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908480 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a7b7ab-3445-4878-ba81-277a85be52c4" containerName="maas-api" Oct 30 20:46:49.908620 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.908491 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0234ef00-f5bc-424f-9ec2-499f63e0c217" containerName="maas-api" Oct 30 20:46:49.913576 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.913554 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:49.930488 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.930460 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-72e6e5-openshift-default-dockercfg-gbrl2\"" Oct 30 20:46:49.950343 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:49.950304 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:46:50.019865 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.019827 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrx4p\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.019939 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.019996 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020020 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020057 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020046 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020152 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020207 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020184 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020209 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.020278 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.020245 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121521 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121569 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121577 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121606 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121628 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121648 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121667 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121737 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrx4p\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.121832 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121808 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121880 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.121995 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122167 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.122035 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.122187 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122377 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.122332 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.122500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.122396 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.124098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.124070 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.124288 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.124269 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.140023 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.139994 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.140300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.140277 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrx4p\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\") pod \"e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.227495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.227450 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:50.536820 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.536793 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:46:50.539393 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:50.539349 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0cd209_fe03_4034_9df6_45ce90417240.slice/crio-3737ed93738be20b162dd84b42ccde9d20b403b9e80862a91e7780a7c19c8b09 WatchSource:0}: Error finding container 3737ed93738be20b162dd84b42ccde9d20b403b9e80862a91e7780a7c19c8b09: Status 404 returned error can't find the container with id 3737ed93738be20b162dd84b42ccde9d20b403b9e80862a91e7780a7c19c8b09 Oct 30 20:46:50.541610 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.541577 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:50.541716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.541658 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:50.541716 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:50.541701 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:50.869321 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:46:50.869232 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:46:51.100353 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:51.100315 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerStarted","Data":"29b9fb25780a01ec7782e2b40db9380da2f40c149d11ecf10f5c6f812d3aa626"} Oct 30 20:46:51.100806 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:51.100385 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerStarted","Data":"3737ed93738be20b162dd84b42ccde9d20b403b9e80862a91e7780a7c19c8b09"} Oct 30 20:46:51.228427 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:51.228395 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:51.233634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:51.233605 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:51.275010 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:51.274950 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podStartSLOduration=2.274934896 podStartE2EDuration="2.274934896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:51.161124047 +0000 UTC m=+4278.610616772" watchObservedRunningTime="2026-10-30 20:46:51.274934896 +0000 UTC m=+4278.724427621" Oct 30 20:46:52.106516 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:52.106487 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:52.107555 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:52.107535 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:46:55.403973 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.403934 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:46:55.409172 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.409148 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.432046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.432002 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-72e6e5-dockercfg-zn2rq\"" Oct 30 20:46:55.432046 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.432016 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-72e6e5\"" Oct 30 20:46:55.434071 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.434044 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:46:55.480752 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.480719 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.480939 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.480779 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwzv\" (UniqueName: \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.504416 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.504354 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:46:55.509040 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.509006 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.536204 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.536174 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:46:55.581855 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.581823 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.582030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.581892 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwzv\" (UniqueName: \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.582030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.581934 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.582030 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.582007 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvfp\" (UniqueName: \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.582602 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.582580 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.611102 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.611067 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwzv\" (UniqueName: \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\") pod \"payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.683609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.683516 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.683609 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.683583 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvfp\" (UniqueName: \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.684333 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.684307 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.707829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.707797 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvfp\" (UniqueName: \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\") pod \"payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:55.722687 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.722656 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:55.826900 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:55.826860 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:56.044239 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.044039 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:46:56.046789 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:56.046763 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054c307_1723_4d3f_b38b_d49191d18292.slice/crio-c4c250ce13bf957bb47033bce3afe3bcc355d1632a41a431c9d252e713aa332c WatchSource:0}: Error finding container c4c250ce13bf957bb47033bce3afe3bcc355d1632a41a431c9d252e713aa332c: Status 404 returned error can't find the container with id c4c250ce13bf957bb47033bce3afe3bcc355d1632a41a431c9d252e713aa332c Oct 30 20:46:56.128318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.128289 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" event={"ID":"8054c307-1723-4d3f-b38b-d49191d18292","Type":"ContainerStarted","Data":"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44"} Oct 30 20:46:56.128477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.128338 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" event={"ID":"8054c307-1723-4d3f-b38b-d49191d18292","Type":"ContainerStarted","Data":"c4c250ce13bf957bb47033bce3afe3bcc355d1632a41a431c9d252e713aa332c"} Oct 30 20:46:56.128477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.128413 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:46:56.149219 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.149192 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:46:56.151699 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:46:56.151658 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca6b79b_83f8_45d1_a2a0_73d1efb13ddf.slice/crio-b431afc9391ae14714d232a71d08c3b49e99a9d31622b216a9a45fbf1c8eb58c WatchSource:0}: Error finding container b431afc9391ae14714d232a71d08c3b49e99a9d31622b216a9a45fbf1c8eb58c: Status 404 returned error can't find the container with id b431afc9391ae14714d232a71d08c3b49e99a9d31622b216a9a45fbf1c8eb58c Oct 30 20:46:56.179168 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:56.179107 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" podStartSLOduration=1.179089196 podStartE2EDuration="1.179089196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:56.178143621 +0000 UTC m=+4283.627636341" watchObservedRunningTime="2026-10-30 20:46:56.179089196 +0000 UTC m=+4283.628581922" Oct 30 20:46:57.136678 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:57.136634 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" event={"ID":"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf","Type":"ContainerStarted","Data":"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86"} Oct 30 20:46:57.137131 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:57.136698 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" event={"ID":"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf","Type":"ContainerStarted","Data":"b431afc9391ae14714d232a71d08c3b49e99a9d31622b216a9a45fbf1c8eb58c"} Oct 30 20:46:57.172737 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:57.172681 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" podStartSLOduration=2.172665619 podStartE2EDuration="2.172665619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:57.171876094 +0000 UTC m=+4284.621368832" watchObservedRunningTime="2026-10-30 20:46:57.172665619 +0000 UTC m=+4284.622158344" Oct 30 20:46:58.143814 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:58.143784 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:46:59.009683 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:46:59.009639 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:47:07.139994 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:07.139961 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:47:08.149582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:08.149536 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:08.150018 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:08.149606 5991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:09.151969 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.151937 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:47:09.231613 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.231582 5991 generic.go:356] "Generic (PLEG): container finished" podID="0c0cd209-fe03-4034-9df6-45ce90417240" containerID="29b9fb25780a01ec7782e2b40db9380da2f40c149d11ecf10f5c6f812d3aa626" exitCode=0 Oct 30 20:47:09.231829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.231632 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerDied","Data":"29b9fb25780a01ec7782e2b40db9380da2f40c149d11ecf10f5c6f812d3aa626"} Oct 30 20:47:09.232096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.232076 5991 scope.go:122] "RemoveContainer" containerID="29b9fb25780a01ec7782e2b40db9380da2f40c149d11ecf10f5c6f812d3aa626" Oct 30 20:47:09.233355 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.233324 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:09.233455 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.233428 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:09.233495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:09.233465 5991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:10.238650 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:10.238614 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerStarted","Data":"044fec63e427c3c1be25108f85009cad4956532b52eb1742fed64f4635973be8"} Oct 30 20:47:11.227896 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:11.227857 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:47:11.228210 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:11.228175 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:47:11.228334 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:11.228232 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 20:47:12.228300 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:12.228264 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:47:12.228747 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:12.228329 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 20:47:13.228831 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:13.228792 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:47:13.229337 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:13.228851 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 20:47:14.228225 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:14.228184 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:47:14.228466 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:14.228245 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 20:47:15.228277 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:15.228234 5991 patch_prober.go:29] interesting pod/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:47:15.228735 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:15.228300 5991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 20:47:16.232645 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:16.232604 5991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:47:16.233161 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:16.233014 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:47:16.233761 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:47:16.233742 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:47:20.869658 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:47:20.869619 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:47:50.869858 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:47:50.869773 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:48:20.869091 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:48:20.869040 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:48:38.972319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:38.972282 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:48:39.000634 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:39.000593 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:48:43.973500 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:43.973447 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" podUID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" containerName="payload-processing" containerID="cri-o://834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86" gracePeriod=25 Oct 30 20:48:44.001409 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.001343 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" podUID="8054c307-1723-4d3f-b38b-d49191d18292" containerName="payload-pre-processing" containerID="cri-o://9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44" gracePeriod=25 Oct 30 20:48:44.256374 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.256330 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:48:44.290017 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.289990 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:48:44.383903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.383864 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\" (UniqueName: \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\") pod \"8054c307-1723-4d3f-b38b-d49191d18292\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " Oct 30 20:48:44.384103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.383931 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\" (UniqueName: \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\") pod \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " Oct 30 20:48:44.384103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.383994 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\") pod \"8054c307-1723-4d3f-b38b-d49191d18292\" (UID: \"8054c307-1723-4d3f-b38b-d49191d18292\") " Oct 30 20:48:44.384103 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.384041 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\") pod \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\" (UID: \"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf\") " Oct 30 20:48:44.384454 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.384429 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume" pod "8054c307-1723-4d3f-b38b-d49191d18292" (UID: "8054c307-1723-4d3f-b38b-d49191d18292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:44.384534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.384475 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume" pod "6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" (UID: "6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:44.386133 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.386114 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp" pod "6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" (UID: "6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf"). InnerVolumeSpecName "kube-api-access-bwvfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:44.386201 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.386138 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv" pod "8054c307-1723-4d3f-b38b-d49191d18292" (UID: "8054c307-1723-4d3f-b38b-d49191d18292"). InnerVolumeSpecName "kube-api-access-nbwzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:44.485402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.485271 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:48:44.485402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.485305 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbwzv\" (UniqueName: \"kubernetes.io/projected/8054c307-1723-4d3f-b38b-d49191d18292-kube-api-access-nbwzv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:48:44.485402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.485318 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvfp\" (UniqueName: \"kubernetes.io/projected/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf-kube-api-access-bwvfp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:48:44.485402 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.485332 5991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054c307-1723-4d3f-b38b-d49191d18292-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:48:44.738049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.737953 5991 generic.go:356] "Generic (PLEG): container finished" podID="8054c307-1723-4d3f-b38b-d49191d18292" containerID="9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44" exitCode=0 Oct 30 20:48:44.738049 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.738005 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" Oct 30 20:48:44.738304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.738043 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" event={"ID":"8054c307-1723-4d3f-b38b-d49191d18292","Type":"ContainerDied","Data":"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44"} Oct 30 20:48:44.738304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.738105 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2" event={"ID":"8054c307-1723-4d3f-b38b-d49191d18292","Type":"ContainerDied","Data":"c4c250ce13bf957bb47033bce3afe3bcc355d1632a41a431c9d252e713aa332c"} Oct 30 20:48:44.738304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.738128 5991 scope.go:122] "RemoveContainer" containerID="9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44" Oct 30 20:48:44.739667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.739639 5991 generic.go:356] "Generic (PLEG): container finished" podID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" containerID="834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86" exitCode=0 Oct 30 20:48:44.739818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.739691 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" Oct 30 20:48:44.739818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.739700 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" event={"ID":"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf","Type":"ContainerDied","Data":"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86"} Oct 30 20:48:44.739818 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.739747 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m" event={"ID":"6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf","Type":"ContainerDied","Data":"b431afc9391ae14714d232a71d08c3b49e99a9d31622b216a9a45fbf1c8eb58c"} Oct 30 20:48:44.747652 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.747624 5991 scope.go:122] "RemoveContainer" containerID="9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44" Oct 30 20:48:44.747998 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:48:44.747976 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44\": container with ID starting with 9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44 not found: ID does not exist" containerID="9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44" Oct 30 20:48:44.748088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.748015 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44"} err="failed to get container status \"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44\": rpc error: code = NotFound desc = could not find container \"9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44\": container with ID starting with 9fbd6f607a09374373a3b41a1e4b684820e0bc00f1f9d046a9d31c13a8d52b44 not found: ID does not exist" Oct 30 20:48:44.748088 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.748045 5991 scope.go:122] "RemoveContainer" containerID="834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86" Oct 30 20:48:44.757317 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.757294 5991 scope.go:122] "RemoveContainer" containerID="834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86" Oct 30 20:48:44.757701 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:48:44.757681 5991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86\": container with ID starting with 834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86 not found: ID does not exist" containerID="834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86" Oct 30 20:48:44.757774 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.757710 5991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86"} err="failed to get container status \"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86\": rpc error: code = NotFound desc = could not find container \"834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86\": container with ID starting with 834fb34b90055b1fb1957453ea505bd60dc764eb3d5d0e722acb0fb684b0cf86 not found: ID does not exist" Oct 30 20:48:44.801255 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.801219 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:48:44.815964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.815890 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-72e6e5-79bdf7b84-v994m"] Oct 30 20:48:44.855306 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.855270 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:48:44.868868 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:44.868702 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-72e6e5-85cbd99695-w4sr2"] Oct 30 20:48:46.677476 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:46.677438 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" path="/var/lib/kubelet/pods/6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf/volumes" Oct 30 20:48:46.677919 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:46.677904 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8054c307-1723-4d3f-b38b-d49191d18292" path="/var/lib/kubelet/pods/8054c307-1723-4d3f-b38b-d49191d18292/volumes" Oct 30 20:48:48.409647 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:48.409601 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:48:50.869316 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:48:50.869278 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:48:54.581497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:54.581455 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:48:54.582024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:54.581775 5991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" containerID="cri-o://044fec63e427c3c1be25108f85009cad4956532b52eb1742fed64f4635973be8" gracePeriod=30 Oct 30 20:48:57.207173 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:57.207139 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:48:59.854850 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:59.854810 5991 generic.go:356] "Generic (PLEG): container finished" podID="0c0cd209-fe03-4034-9df6-45ce90417240" containerID="044fec63e427c3c1be25108f85009cad4956532b52eb1742fed64f4635973be8" exitCode=0 Oct 30 20:48:59.855256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:59.854885 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerDied","Data":"044fec63e427c3c1be25108f85009cad4956532b52eb1742fed64f4635973be8"} Oct 30 20:48:59.855256 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:48:59.854965 5991 scope.go:122] "RemoveContainer" containerID="29b9fb25780a01ec7782e2b40db9380da2f40c149d11ecf10f5c6f812d3aa626" Oct 30 20:49:00.143991 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.143962 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:49:00.303130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303095 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303130 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303136 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303167 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303184 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303219 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303262 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303288 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303388 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303358 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303425 5991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\") pod \"0c0cd209-fe03-4034-9df6-45ce90417240\" (UID: \"0c0cd209-fe03-4034-9df6-45ce90417240\") " Oct 30 20:49:00.303742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303449 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:49:00.303742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303520 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:49:00.303742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303548 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:49:00.303981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303799 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:49:00.303981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303799 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:49:00.303981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303881 5991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.303981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303902 5991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.303981 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.303918 5991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.305697 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.305669 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:49:00.306203 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.306185 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "kube-api-access-nrx4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:49:00.306292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.306205 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:49:00.306292 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.306234 5991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy" pod "0c0cd209-fe03-4034-9df6-45ce90417240" (UID: "0c0cd209-fe03-4034-9df6-45ce90417240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404813 5991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cd209-fe03-4034-9df6-45ce90417240-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404849 5991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404863 5991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd209-fe03-4034-9df6-45ce90417240-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404872 5991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cd209-fe03-4034-9df6-45ce90417240-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404880 5991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrx4p\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-kube-api-access-nrx4p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.404903 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.404890 5991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cd209-fe03-4034-9df6-45ce90417240-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\"" Oct 30 20:49:00.862259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.862223 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" event={"ID":"0c0cd209-fe03-4034-9df6-45ce90417240","Type":"ContainerDied","Data":"3737ed93738be20b162dd84b42ccde9d20b403b9e80862a91e7780a7c19c8b09"} Oct 30 20:49:00.862259 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.862258 5991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg" Oct 30 20:49:00.862762 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.862283 5991 scope.go:122] "RemoveContainer" containerID="044fec63e427c3c1be25108f85009cad4956532b52eb1742fed64f4635973be8" Oct 30 20:49:00.919134 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.919094 5991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:49:00.923028 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:00.923000 5991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-72e6e5-openshift-default-77bbbb6789-fpfbg"] Oct 30 20:49:02.676537 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:02.676497 5991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" path="/var/lib/kubelet/pods/0c0cd209-fe03-4034-9df6-45ce90417240/volumes" Oct 30 20:49:05.316829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:05.316795 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:49:20.867438 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:49:20.867331 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:49:23.313045 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:23.313000 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:49:50.868771 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:49:50.868732 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:49:51.308134 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:49:51.308097 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4whww"] Oct 30 20:50:02.539675 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:02.539637 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lxcg9_2860740e-daff-4305-afaf-262de7a22bee/manager/0.log" Oct 30 20:50:03.290400 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:03.290354 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rlcr5_589ad651-f4dd-484b-92c0-5179357485bf/server/0.log" Oct 30 20:50:04.441588 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:04.441552 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f6f44c6f-72rhq_5e90c01f-09e5-4dff-a6a4-2a25e056db24/maas-api/0.log" Oct 30 20:50:08.163986 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:08.163954 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mg72b_b1bda3e4-bb2b-4269-9ddf-0c536750dfc1/kuadrant-console-plugin/0.log" Oct 30 20:50:08.600786 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:08.600757 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4whww_76bafc1c-67f0-4b74-b191-6484890bbc1d/limitador/0.log" Oct 30 20:50:09.983431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:09.983401 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-mlfq7_f6839fdf-2c95-4b36-9919-eac9d385123a/kube-auth-proxy/0.log" Oct 30 20:50:10.378838 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:10.378753 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-77bmk_5386bfdc-a1aa-40ec-b5de-2481f63e5184/istio-proxy/0.log" Oct 30 20:50:11.748582 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:11.748550 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-psrq4_c9fda10e-bbc9-4010-b06d-a2c2d7427f96/istio-proxy/0.log" Oct 30 20:50:11.952829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:11.952792 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rgnvj_f94a0aec-c628-48a5-bd9c-8483df0281fb/istio-proxy/0.log" Oct 30 20:50:12.735183 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:12.735142 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:50:13.909024 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:13.908996 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw_ab0d2e72-c6c8-482c-b69f-9fd413e01569/main/0.log" Oct 30 20:50:14.307650 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:14.307620 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-bf7cf_47a539a2-c29a-47c4-93bd-ebc74ad85856/main/0.log" Oct 30 20:50:14.492477 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:14.492448 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-td6ls_789b903b-be64-4bdd-bf2c-20dd94fbce79/main/0.log" Oct 30 20:50:14.877742 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:14.877708 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ngh9x_45d0120c-b448-490e-ae2b-040abdd254c3/main/0.log" Oct 30 20:50:15.087353 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:15.087303 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq_62b90662-f37f-4ba0-a9fc-9b37bcdfe838/main/0.log" Oct 30 20:50:15.288982 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:15.288948 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh_0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36/main/0.log" Oct 30 20:50:20.868358 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:50:20.868317 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:50:25.910920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:25.910885 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lxcg9_2860740e-daff-4305-afaf-262de7a22bee/manager/0.log" Oct 30 20:50:26.562318 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:26.562278 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rlcr5_589ad651-f4dd-484b-92c0-5179357485bf/server/0.log" Oct 30 20:50:27.484242 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:27.484203 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f6f44c6f-72rhq_5e90c01f-09e5-4dff-a6a4-2a25e056db24/maas-api/0.log" Oct 30 20:50:30.679151 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:30.679119 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mg72b_b1bda3e4-bb2b-4269-9ddf-0c536750dfc1/kuadrant-console-plugin/0.log" Oct 30 20:50:31.103720 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:31.103685 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4whww_76bafc1c-67f0-4b74-b191-6484890bbc1d/limitador/0.log" Oct 30 20:50:32.263495 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:32.263464 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-mlfq7_f6839fdf-2c95-4b36-9919-eac9d385123a/kube-auth-proxy/0.log" Oct 30 20:50:32.650920 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:32.650831 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-77bmk_5386bfdc-a1aa-40ec-b5de-2481f63e5184/istio-proxy/0.log" Oct 30 20:50:33.442413 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:33.442358 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:50:33.479443 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:33.479414 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5fx8_908aaccc-1937-4ac9-8614-28365a9fb984/ovn-acl-logging/0.log" Oct 30 20:50:34.050808 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:34.050771 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-psrq4_c9fda10e-bbc9-4010-b06d-a2c2d7427f96/istio-proxy/0.log" Oct 30 20:50:34.246189 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:34.246158 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rgnvj_f94a0aec-c628-48a5-bd9c-8483df0281fb/istio-proxy/0.log" Oct 30 20:50:35.025497 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:35.025467 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:50:35.752176 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:35.752150 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j4ggw_ab0d2e72-c6c8-482c-b69f-9fd413e01569/main/0.log" Oct 30 20:50:36.175668 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:36.175568 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-bf7cf_47a539a2-c29a-47c4-93bd-ebc74ad85856/main/0.log" Oct 30 20:50:36.378704 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:36.378671 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-td6ls_789b903b-be64-4bdd-bf2c-20dd94fbce79/main/0.log" Oct 30 20:50:36.784766 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:36.784732 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ngh9x_45d0120c-b448-490e-ae2b-040abdd254c3/main/0.log" Oct 30 20:50:36.988623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:36.988593 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2kwhq_62b90662-f37f-4ba0-a9fc-9b37bcdfe838/main/0.log" Oct 30 20:50:37.187320 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:50:37.187232 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7hgqh_0ce8e3d2-4c44-4d20-8f68-0bd9b5a83c36/main/0.log" Oct 30 20:50:50.867968 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:50:50.867888 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:51:13.272772 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:51:13.272740 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:51:13.479132 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:51:13.479102 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:51:20.868594 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:51:20.868554 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:51:50.867959 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:51:50.867916 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:51:57.294160 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:51:57.294122 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:51:57.482824 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:51:57.482795 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log" Oct 30 20:52:02.668667 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:02.668632 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zzgh7_92a56f5b-251a-4f17-9dc8-ed6434db1328/global-pull-secret-syncer/0.log" Oct 30 20:52:03.368144 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:03.368109 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l7lfb_bfb65a5a-630a-4ad4-b6a8-2bd89928ac9e/konnectivity-agent/0.log" Oct 30 20:52:03.875136 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:03.875099 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-4.ec2.internal_a66c9af704a53c4c9b04653e0d8d8d8f/haproxy/0.log" Oct 30 20:52:11.069275 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:11.069247 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mg72b_b1bda3e4-bb2b-4269-9ddf-0c536750dfc1/kuadrant-console-plugin/0.log" Oct 30 20:52:11.356928 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:11.356827 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4whww_76bafc1c-67f0-4b74-b191-6484890bbc1d/limitador/0.log" Oct 30 20:52:12.447126 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447085 5991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq"] Oct 30 20:52:12.447767 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447748 5991 cpu_manager.go:414] "containerMap: removing container" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" Oct 30 20:52:12.447826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447769 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" Oct 30 20:52:12.447826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447778 5991 cpu_manager.go:414] "containerMap: removing container" podUID="8054c307-1723-4d3f-b38b-d49191d18292" containerName="payload-pre-processing" Oct 30 20:52:12.447826 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447784 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8054c307-1723-4d3f-b38b-d49191d18292" containerName="payload-pre-processing" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447830 5991 cpu_manager.go:414] "containerMap: removing container" podUID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" containerName="payload-processing" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447835 5991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" containerName="payload-processing" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447903 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447913 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca6b79b-83f8-45d1-a2a0-73d1efb13ddf" containerName="payload-processing" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447921 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0cd209-fe03-4034-9df6-45ce90417240" containerName="istio-proxy" Oct 30 20:52:12.447941 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.447926 5991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8054c307-1723-4d3f-b38b-d49191d18292" containerName="payload-pre-processing" Oct 30 20:52:12.451116 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.451096 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.463828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.463802 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"kube-root-ca.crt\"" Oct 30 20:52:12.474166 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.474138 5991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7snw\"/\"default-dockercfg-965f6\"" Oct 30 20:52:12.474331 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.474138 5991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"openshift-service-ca.crt\"" Oct 30 20:52:12.476078 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.476054 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq"] Oct 30 20:52:12.536721 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.536684 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-podres\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.536936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.536748 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-lib-modules\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.536936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.536865 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-proc\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.536936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.536894 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-sys\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.536936 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.536930 5991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78h82\" (UniqueName: \"kubernetes.io/projected/df996eed-95e0-41e0-9780-408af200c550-kube-api-access-78h82\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638319 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638281 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78h82\" (UniqueName: \"kubernetes.io/projected/df996eed-95e0-41e0-9780-408af200c550-kube-api-access-78h82\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638426 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-podres\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638483 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-lib-modules\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638516 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-proc\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638534 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638533 5991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-sys\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638564 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-podres\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638603 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-sys\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638630 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-lib-modules\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.638731 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.638653 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df996eed-95e0-41e0-9780-408af200c550-proc\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.656188 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.656158 5991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78h82\" (UniqueName: \"kubernetes.io/projected/df996eed-95e0-41e0-9780-408af200c550-kube-api-access-78h82\") pod \"perf-node-gather-daemonset-5wtzq\" (UID: \"df996eed-95e0-41e0-9780-408af200c550\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:12.765199 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:12.765153 5991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:13.046150 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.046074 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/alertmanager/0.log" Oct 30 20:52:13.058080 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.058049 5991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq"] Oct 30 20:52:13.061820 ip-10-0-132-4 kubenswrapper[5991]: W1030 20:52:13.061793 5991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf996eed_95e0_41e0_9780_408af200c550.slice/crio-526f630c7c96d5f43d8d2d459f6bd7523d4a831477575d6fe54d6fbe0ef87370 WatchSource:0}: Error finding container 526f630c7c96d5f43d8d2d459f6bd7523d4a831477575d6fe54d6fbe0ef87370: Status 404 returned error can't find the container with id 526f630c7c96d5f43d8d2d459f6bd7523d4a831477575d6fe54d6fbe0ef87370 Oct 30 20:52:13.166286 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.166257 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/config-reloader/0.log" Oct 30 20:52:13.286489 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.286454 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/kube-rbac-proxy-web/0.log" Oct 30 20:52:13.407895 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.407816 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/kube-rbac-proxy/0.log" Oct 30 20:52:13.527928 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.527896 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/kube-rbac-proxy-metric/0.log" Oct 30 20:52:13.652431 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.652403 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/prom-label-proxy/0.log" Oct 30 20:52:13.771193 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.771162 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df553a58-0c71-4068-9409-f67d91ad6150/init-config-reloader/0.log" Oct 30 20:52:13.948152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.948119 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" event={"ID":"df996eed-95e0-41e0-9780-408af200c550","Type":"ContainerStarted","Data":"82ebe538e2f00d252802b2833bec4e59ac3fc05275671df585f5559958666678"} Oct 30 20:52:13.948358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.948173 5991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" event={"ID":"df996eed-95e0-41e0-9780-408af200c550","Type":"ContainerStarted","Data":"526f630c7c96d5f43d8d2d459f6bd7523d4a831477575d6fe54d6fbe0ef87370"} Oct 30 20:52:13.948358 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.948280 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:13.984076 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:13.984021 5991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" podStartSLOduration=1.984006673 podStartE2EDuration="1.984006673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:13.982700585 +0000 UTC m=+4601.432193312" watchObservedRunningTime="2026-10-30 20:52:13.984006673 +0000 UTC m=+4601.433499396" Oct 30 20:52:14.028964 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.028829 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t5rks_724b9880-4314-41ad-93f7-40eb026a0fb2/kube-state-metrics/0.log" Oct 30 20:52:14.138876 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.138845 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t5rks_724b9880-4314-41ad-93f7-40eb026a0fb2/kube-rbac-proxy-main/0.log" Oct 30 20:52:14.258623 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.258595 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t5rks_724b9880-4314-41ad-93f7-40eb026a0fb2/kube-rbac-proxy-self/0.log" Oct 30 20:52:14.382098 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.382022 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-695b5bb69b-wlqqm_99ab51f7-f13e-41df-985a-172ac7fe4927/metrics-server/0.log" Oct 30 20:52:14.504726 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.504693 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-dvj8z_d4063bcb-d0a9-4bab-89a1-fe44f1a9e0e7/monitoring-plugin/0.log" Oct 30 20:52:14.630840 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.630809 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csl8b_d5ecb26d-be82-4455-8dae-e0b7489350ee/node-exporter/0.log" Oct 30 20:52:14.749304 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.749272 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csl8b_d5ecb26d-be82-4455-8dae-e0b7489350ee/kube-rbac-proxy/0.log" Oct 30 20:52:14.868535 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:14.868506 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csl8b_d5ecb26d-be82-4455-8dae-e0b7489350ee/init-textfile/0.log" Oct 30 20:52:15.728262 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:15.728230 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6m25j_ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/kube-rbac-proxy-main/0.log" Oct 30 20:52:15.847599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:15.847569 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6m25j_ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/kube-rbac-proxy-self/0.log" Oct 30 20:52:15.967622 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:15.967590 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6m25j_ff8b2a9b-294c-4563-8b7e-6c6fa3d621ba/openshift-state-metrics/0.log" Oct 30 20:52:16.100980 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.100891 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/prometheus/0.log" Oct 30 20:52:16.216828 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.216800 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/config-reloader/0.log" Oct 30 20:52:16.334966 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.334936 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/thanos-sidecar/0.log" Oct 30 20:52:16.453472 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.453397 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/kube-rbac-proxy-web/0.log" Oct 30 20:52:16.574128 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.574099 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/kube-rbac-proxy/0.log" Oct 30 20:52:16.700744 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.700716 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/kube-rbac-proxy-thanos/0.log" Oct 30 20:52:16.820648 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:16.820621 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a6a8e960-a61f-4dfa-9ad7-dbc6c7442873/init-config-reloader/0.log" Oct 30 20:52:17.330206 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.330177 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84748fcdfb-kqjdr_dc76c3f5-d680-4468-a8ae-b100e41e253c/telemeter-client/0.log" Oct 30 20:52:17.450760 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.450728 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84748fcdfb-kqjdr_dc76c3f5-d680-4468-a8ae-b100e41e253c/reload/0.log" Oct 30 20:52:17.569080 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.569040 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84748fcdfb-kqjdr_dc76c3f5-d680-4468-a8ae-b100e41e253c/kube-rbac-proxy/0.log" Oct 30 20:52:17.696540 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.696434 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/thanos-query/0.log" Oct 30 20:52:17.813054 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.813021 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/kube-rbac-proxy-web/0.log" Oct 30 20:52:17.935425 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:17.935395 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/kube-rbac-proxy/0.log" Oct 30 20:52:18.055164 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:18.055131 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/prom-label-proxy/0.log" Oct 30 20:52:18.175328 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:18.175297 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/kube-rbac-proxy-rules/0.log" Oct 30 20:52:18.292750 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:18.292697 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747576895b-zqwt6_e7977864-d9ee-43d9-9c44-6611a31dba60/kube-rbac-proxy-metrics/0.log" Oct 30 20:52:19.964684 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:19.964652 5991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-5wtzq" Oct 30 20:52:20.869706 ip-10-0-132-4 kubenswrapper[5991]: E1030 20:52:20.869659 5991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ts96r" containerName="extractor" Oct 30 20:52:26.227152 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:26.227118 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgmdj_98c38cf2-9e21-4d7f-836b-9983359012fc/dns/0.log" Oct 30 20:52:26.345599 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:26.345559 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgmdj_98c38cf2-9e21-4d7f-836b-9983359012fc/kube-rbac-proxy/0.log" Oct 30 20:52:26.720829 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:26.720738 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmsqs_c58f03ba-4840-4237-88ee-82bf233ac315/dns-node-resolver/0.log" Oct 30 20:52:28.350014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:28.349980 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftqg8_3ac523a2-773b-493f-bb76-8faa6f8c3bb1/node-ca/0.log" Oct 30 20:52:28.744014 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:28.743981 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-mlfq7_f6839fdf-2c95-4b36-9919-eac9d385123a/kube-auth-proxy/0.log" Oct 30 20:52:28.878096 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:28.878063 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-77bmk_5386bfdc-a1aa-40ec-b5de-2481f63e5184/istio-proxy/0.log" Oct 30 20:52:29.341651 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:29.341618 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-psrq4_c9fda10e-bbc9-4010-b06d-a2c2d7427f96/istio-proxy/0.log" Oct 30 20:52:29.412147 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:29.412111 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rgnvj_f94a0aec-c628-48a5-bd9c-8483df0281fb/istio-proxy/0.log" Oct 30 20:52:29.676690 ip-10-0-132-4 kubenswrapper[5991]: I1030 20:52:29.676590 5991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6vn57_3d3a5c43-a952-42f9-9e67-3a629aca6b72/payload-processing/0.log"